pydiverse/pydiverse.pipedag issues and pull requests

#199 - Mixed per-user and team-shared pipeline runs

Issue - State: open - Opened by windiana42 25 days ago

#198 - Solve connectorx failure

Pull Request - State: closed - Opened by windiana42 29 days ago

#197 - fix(): add previous_hook_replace argument for register_table

Pull Request - State: closed - Opened by DominikZuercherQC 30 days ago

#196 - feat(): add upload_table and download_table functions to PandasTableHook

Pull Request - State: closed - Opened by DominikZuercherQC about 1 month ago

#195 - Persist manually versioned intermediate state / decoupling development on two parts of the pipeline

Issue - State: open - Opened by windiana42 about 1 month ago
Labels: enhancement, usability

#194 - Support calling subgraph even if position hashes of inputs changed

Issue - State: open - Opened by windiana42 about 2 months ago
Labels: better debugging, usability

#193 - Improve error message for input not found when running subgraph

Issue - State: open - Opened by windiana42 about 2 months ago
Labels: bug, better debugging, usability

#192 - Allow running tasks on active schema (Big DANGER mode disclaimer)

Issue - State: open - Opened by windiana42 about 2 months ago
Labels: enhancement, usability, speed

#191 - DuckDB: Fix Table rename with primary key/index

Issue - State: open - Opened by windiana42 about 2 months ago
Labels: bug

#190 - Move primary key generation for mssql backend after completely filling a table.

Pull Request - State: closed - Opened by windiana42 2 months ago

#189 - Implement bulk load to snowflake database (the driver has a mechanism to load parquet files via S3)

Issue - State: open - Opened by windiana42 2 months ago
Labels: usability, speed

#188 - Implement @input_stage_version decorator which provides inputs from two versions of the same stage before schema swapping

Pull Request - State: closed - Opened by windiana42 2 months ago

#187 - Better error message when executing subflow and input tasks not found in cache

Issue - State: open - Opened by windiana42 2 months ago
Labels: better debugging, usability, error messages

#186 - Implementation of snowflake backend.

Pull Request - State: closed - Opened by windiana42 3 months ago - 2 comments

#185 - Do not require connectorx for polars dematerialization

Pull Request - State: closed - Opened by nicolasmueller 3 months ago

#184 - Support Snowflake backend

Issue - State: closed - Opened by windiana42 3 months ago - 1 comment
Labels: enhancement

#183 - Watch duckdb 0.10.1 and sqlalchemy 2.0.29 compatibility issues

Issue - State: open - Opened by windiana42 3 months ago - 1 comment
Labels: documentation, dependencies, blocked external

#182 - Add group nodes for structuring flows, as task ordering barrier, and for clusting tasks in visualization

Pull Request - State: closed - Opened by windiana42 3 months ago - 5 comments

#181 - Add validation tasks to a Stage which get access to multiple schemas at once (last committed schema of this instance or another instance for this Stage)

Issue - State: closed - Opened by windiana42 3 months ago - 8 comments
Labels: enhancement, usability

#180 - Group tasks or stages in visualization

Issue - State: closed - Opened by windiana42 3 months ago - 5 comments
Labels: enhancement, usability

#179 - Implement imperative materialization.

Pull Request - State: closed - Opened by windiana42 3 months ago

#178 - Improve insert_into_in_query to treat comments in TSQL select statements

Issue - State: open - Opened by windiana42 3 months ago - 1 comment
Labels: good first issue, reliability, usability

#177 - fix(): Column called from triggers bug

Pull Request - State: closed - Opened by DominikZuercherQC 3 months ago

#176 - add autoincrement option

Pull Request - State: open - Opened by DominikZuercherQC 3 months ago

#175 - Provide a lot more examples for readthedocs.io documentation

Pull Request - State: closed - Opened by windiana42 3 months ago - 1 comment

#174 - Provide pydiverse.pipedag.version

Issue - State: open - Opened by windiana42 3 months ago
Labels: good first issue, usability

#173 - Move some or all combinatorial tests for cache_validation options to nightly tests.

Issue - State: open - Opened by windiana42 3 months ago
Labels: testing, speed

#172 - Add identity_insert materialisation option for MSSQL tablestores

Pull Request - State: open - Opened by DominikZuercherQC 3 months ago

#171 - Setting the isolation level for MSSQL table store to READ UNCOMMITED

Pull Request - State: closed - Opened by DominikZuercherQC 3 months ago

#170 - Propose redesign of cache validation options in configuration

Pull Request - State: closed - Opened by windiana42 3 months ago - 5 comments

#169 - mssql: Fast materialization/dematerialization based on bcpandas and similar bulk load techniques

Issue - State: open - Opened by windiana42 3 months ago
Labels: speed

#168 - Emergency Release 0.7.2: Implemented config for max_copy_operations, pool_size, pool_timeout.

Pull Request - State: closed - Opened by windiana42 3 months ago

#167 - Retry of producing a stage output

Issue - State: open - Opened by windiana42 4 months ago
Labels: enhancement, usability, refactoring

#166 - Add option to `Flow.run` to force execution of all tasks

Pull Request - State: closed - Opened by nicolasmueller 4 months ago - 5 comments

#165 - Disable cache validation for a specific run

Issue - State: closed - Opened by nicolasmueller 4 months ago
Labels: usability

#164 - Imperative Materialization

Issue - State: closed - Opened by windiana42 4 months ago - 2 comments
Labels: enhancement, usability, refactoring

#163 - Implement inline feature in pipedag.Table() to prevent materialization to database if not needed

Issue - State: open - Opened by windiana42 4 months ago - 2 comments
Labels: enhancement, usability, refactoring, speed

#162 - Fix #127 regarding UNLOGGED postgres tables during copy table.

Pull Request - State: closed - Opened by windiana42 4 months ago - 1 comment

#161 - Implement systematic retry logic especially for `sa.Table(autoload_with=)`

Issue - State: open - Opened by windiana42 4 months ago - 1 comment
Labels: partially done, testing, reliability, usability

#160 - Support nullable and non_nullable parameters in pipedag.Table() and refactor ddl/table-store/hook logic

Pull Request - State: closed - Opened by windiana42 4 months ago - 2 comments

#159 - Support setting nullable / non-nullable constraints in pipedag.Table()

Issue - State: closed - Opened by windiana42 4 months ago - 6 comments
Labels: enhancement, usability

#158 - Refactor DB2 dependency in generic code.

Pull Request - State: closed - Opened by windiana42 4 months ago - 2 comments

#157 - Fix duplicate table bug

Pull Request - State: closed - Opened by nicolasmueller 4 months ago

#156 - Disable kroki links by default.

Pull Request - State: closed - Opened by windiana42 4 months ago - 1 comment

#155 - Adding max_nchars_query_print option for MSSqlTableStore

Pull Request - State: closed - Opened by DominikZuercherQC 4 months ago - 2 comments

#154 - Smallish DDL extension

Pull Request - State: closed - Opened by windiana42 4 months ago

#153 - Fix numeric to float

Pull Request - State: closed - Opened by windiana42 4 months ago

#152 - Fix duckdb example documentation

Pull Request - State: closed - Opened by windiana42 4 months ago - 2 comments

#151 - Simplify logging inside tasks

Issue - State: open - Opened by windiana42 4 months ago
Labels: usability

#150 - Automatically add ExternalTableReference to auto_table list

Issue - State: open - Opened by windiana42 4 months ago - 1 comment
Labels: usability

#149 - Fix Polars and IBIS tests with DB2 URL and connectorX

Issue - State: open - Opened by windiana42 4 months ago - 1 comment
Labels: dependencies, reliability, usability

#148 - #142: Easier configuration setup

Pull Request - State: closed - Opened by windiana42 4 months ago - 3 comments

#147 - Add some unicode tests

Pull Request - State: closed - Opened by windiana42 4 months ago

#146 - Make max_length = 5000 configurable in mssql dialect for RawSQL Strings

Issue - State: closed - Opened by QuantCo-mr-T 4 months ago - 1 comment
Labels: good first issue, usability

#145 - Update link to table backends

Pull Request - State: closed - Opened by mohahf19 4 months ago

#144 - Fix task_context.input_tables (needed for DB2 input table locking)

Pull Request - State: closed - Opened by nicolasmueller 4 months ago

#143 - Documented input_type options and concepts for future enhancements.

Pull Request - State: closed - Opened by windiana42 4 months ago

#142 - Enable PipedagConfig construction from dictionary

Issue - State: closed - Opened by windiana42 4 months ago - 2 comments
Labels: usability

#141 - Check cache validity of lazy DataFrame tasks.

Issue - State: open - Opened by NicolasMuellerQC 4 months ago
Labels: good first issue, speed

#140 - Make lazy task output tables which do not provide a query string cache invalid for subsequent reading tasks

Pull Request - State: closed - Opened by nicolasmueller 4 months ago - 2 comments

#139 - Enable the creation of external TableReferences

Pull Request - State: closed - Opened by nicolasmueller 4 months ago - 6 comments

#138 - Fix `deep_map`

Pull Request - State: open - Opened by nicolasmueller 4 months ago - 6 comments

#137 - Copy task inputs to protect against modifications before execution time

Pull Request - State: open - Opened by nicolasmueller 5 months ago - 3 comments

#136 - Bump pre-commit/action from 3.0.0 to 3.0.1

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 1 comment
Labels: dependencies

#135 - Use `polars-lts-cpu` instead of `polars` for x86_64 OS X

Pull Request - State: closed - Opened by nicolasmueller 5 months ago - 1 comment
Labels: bug, dependencies

#134 - make pytz and pytzdata optional dependencies

Pull Request - State: closed - Opened by windiana42 5 months ago

#133 - Make pytz optional dependency due to prefect

Pull Request - State: closed - Opened by windiana42 5 months ago - 1 comment

#132 - Reactivate date tests for duckdb

Issue - State: open - Opened by windiana42 5 months ago
Labels: testing, blocked external

#131 - Fix problems in tests after updating dependencies

Pull Request - State: closed - Opened by windiana42 5 months ago

#130 - Document SQLTableStore parameters in config.md

Pull Request - State: open - Opened by nicolasmueller 7 months ago

#129 - Bump actions/setup-python from 4 to 5

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 1 comment
Labels: dependencies

#128 - Fix no materialization details

Pull Request - State: closed - Opened by nicolasmueller 7 months ago

#127 - Support for Postgres unlogged tables when copying tables

Issue - State: closed - Opened by nicolasmueller 7 months ago - 1 comment
Labels: bug

#126 - Support for Delta Loads and Table Partitioning

Issue - State: open - Opened by NicholasHoernleQC 7 months ago - 2 comments
Labels: enhancement

#125 - Postgres unlogged tables as a materialization detail

Pull Request - State: closed - Opened by nicolasmueller 7 months ago

#124 - DB2 RUNSTATS

Pull Request - State: closed - Opened by nicolasmueller 7 months ago

#123 - Instructions for getting pyodbc to run on OS X

Pull Request - State: closed - Opened by nicolasmueller 8 months ago

#122 - Materialization details

Pull Request - State: closed - Opened by nicolasmueller 8 months ago - 4 comments

#121 - Drop DB2 nicknames

Pull Request - State: closed - Opened by nicolasmueller 8 months ago

#120 - Update PR template

Pull Request - State: closed - Opened by nicolasmueller 8 months ago

#120 - Update PR template

Pull Request - State: closed - Opened by nicolasmueller 8 months ago

#119 - DB2 development on OS X arm64

Pull Request - State: closed - Opened by nicolasmueller 8 months ago - 2 comments

#119 - DB2 development on OS X arm64

Pull Request - State: closed - Opened by nicolasmueller 8 months ago - 2 comments

#118 - [Fix] Cascade Schema dropping in IBM DB2

Issue - State: closed - Opened by MichalHeydelQC 8 months ago - 2 comments

#117 - Add a function to ad-hoc materialize a table for interactive debugging

Pull Request - State: closed - Opened by nicolasmueller 8 months ago

#117 - Add a function to ad-hoc materialize a table for interactive debugging

Pull Request - State: closed - Opened by nicolasmueller 8 months ago

#116 - Simplify debugging of materialization write-back

Issue - State: open - Opened by windiana42 8 months ago - 1 comment
Labels: partially done

#116 - Simplify debugging of materialization write-back

Issue - State: open - Opened by windiana42 8 months ago - 1 comment
Labels: partially done

#115 - Create "virtual" stacktrace which combines wiring time and task execution time

Issue - State: open - Opened by windiana42 10 months ago - 2 comments
Labels: enhancement, better debugging

#114 - pipedag-manage delete-schemas: support clearing unused schemas

Issue - State: open - Opened by windiana42 10 months ago
Labels: enhancement, usability

#114 - pipedag-manage delete-schemas: support clearing unused schemas

Issue - State: open - Opened by windiana42 10 months ago
Labels: enhancement, usability

#113 - for DB2 database: allow configuring table creation properties globally and/or per schema or table

Issue - State: closed - Opened by windiana42 10 months ago - 1 comment
Labels: usability, speed

#112 - for DB2 database: RUNSTAT for every table after it was filled and indexes created

Issue - State: closed - Opened by windiana42 10 months ago - 2 comments
Labels: reliability, usability, speed

#111 - Bump actions/checkout from 3 to 4

Pull Request - State: closed - Opened by dependabot[bot] 10 months ago - 1 comment
Labels: dependencies

#110 - Apply database default case to table names during materialization

Issue - State: open - Opened by windiana42 10 months ago - 1 comment
Labels: usability

#110 - Apply database default case to table names during materialization

Issue - State: open - Opened by windiana42 10 months ago - 1 comment
Labels: usability

#109 - Disable warning that parquet table cache cannot dematerialize sa.Table

Issue - State: open - Opened by windiana42 10 months ago - 1 comment
Labels: good first issue, usability, error messages

#108 - Improve error message when polars frame is returned but connectorx is not installed

Issue - State: closed - Opened by windiana42 11 months ago

#108 - Improve error message when polars frame is returned but connectorx is not installed

Issue - State: closed - Opened by windiana42 11 months ago

#107 - Making `ignore_fresh_input` more explicit (or the docs)

Issue - State: closed - Opened by MarcAntoineSchmidtQC 11 months ago - 1 comment

GitHub / pydiverse/pydiverse.pipedag issues and pull requests