pydiverse/pydiverse.pipedag issues and pull requests

#221 - feat(): Release 0.9.7

Pull Request - State: open - Opened by DominikZuercherQC 30 days ago

#220 - feat(): Add support for combination of TaskGetItem and constant inputs

Pull Request - State: closed - Opened by DominikZuercherQC about 1 month ago

#219 - Bump pypa/gh-action-pypi-publish from 1.9.0 to 1.10.0 in the gh-actions group

Pull Request - State: open - Opened by dependabot[bot] about 1 month ago
Labels: dependencies

#218 - Fix and activate snowflake tests in CI

Issue - State: open - Opened by windiana42 about 1 month ago

#217 - Automatic extraction of pipeline subgraph for PROD execution

Issue - State: open - Opened by windiana42 about 2 months ago
Labels: enhancement, usability

#216 - feat(): Task links

Pull Request - State: closed - Opened by DominikZuercherQC about 2 months ago - 2 comments

#215 - Fix pytest assert enrichment problem when running pytest with `--workers` option

Issue - State: open - Opened by windiana42 about 2 months ago
Labels: bug, testing, better debugging, usability

#214 - fix: Only warn about ignore_position_hashes if it is True

Pull Request - State: closed - Opened by nicolasmueller about 2 months ago

#213 - Test in CI for macOS and Windows as well

Issue - State: open - Opened by pavelzw 2 months ago - 1 comment
Labels: testing

#212 - Fix polars >= 1

Pull Request - State: closed - Opened by nicolasmueller 2 months ago

#211 - Implemented ExternalTableReference at flow wiring time.

Pull Request - State: closed - Opened by windiana42 2 months ago

#210 - Fix primary_key generation when writing pandas dataframe to postgres database

Pull Request - State: closed - Opened by windiana42 3 months ago

#209 - Make `MAX_LENGTH_PK` and `MAX_LENGTH_INDEX` configurable

Issue - State: open - Opened by DominikZuercherQC 3 months ago
Labels: good first issue

#208 - Support more table stores (i.e. for parquet files or apache arrow dataframes in-memory)

Issue - State: open - Opened by adzcai 3 months ago - 1 comment
Labels: enhancement

#207 - Switch from poetry (pypi) to pixi (conda-forge)

Pull Request - State: closed - Opened by windiana42 3 months ago - 2 comments

#206 - SQLAlchemy >= 2.0.0, Depedency for IBM-DB2

Issue - State: open - Opened by DelongChenQC 3 months ago

#205 - Batched processing of dataframe based task: reduce memory consumption and parallelize

Issue - State: open - Opened by windiana42 3 months ago - 3 comments
Labels: enhancement, usability, speed

#204 - Complex tasks with imperative materialization break caching

Issue - State: closed - Opened by DominikZuercherQC 3 months ago
Labels: bug

#203 - bug(): Complex tasks using imperative materialization break caching + release `v 0.9.4`

Pull Request - State: closed - Opened by DominikZuercherQC 3 months ago - 2 comments

#202 - feat(): Added ignore_position_hash option for SQLTableStore

Pull Request - State: closed - Opened by DominikZuercherQC 4 months ago - 2 comments

#201 - feat(): add truncation of primary keys and indexes

Pull Request - State: closed - Opened by DominikZuercherQC 4 months ago

#200 - Allow custom identifier names

Issue - State: closed - Opened by DominikZuercherQC 4 months ago
Labels: reliability, usability

#199 - Mixed per-user and team-shared pipeline runs

Issue - State: open - Opened by windiana42 4 months ago
Labels: enhancement, usability, speed

#198 - Solve connectorx failure

Pull Request - State: closed - Opened by windiana42 4 months ago

#197 - fix(): add previous_hook_replace argument for register_table

Pull Request - State: closed - Opened by DominikZuercherQC 4 months ago

#196 - feat(): add upload_table and download_table functions to PandasTableHook

Pull Request - State: closed - Opened by DominikZuercherQC 4 months ago

#195 - Persist manually versioned intermediate state / decoupling development on two parts of the pipeline

Issue - State: open - Opened by windiana42 4 months ago
Labels: enhancement, usability

#194 - Support calling subgraph even if position hashes of inputs changed

Issue - State: closed - Opened by windiana42 5 months ago
Labels: better debugging, usability

#193 - Improve error message for input not found when running subgraph

Issue - State: open - Opened by windiana42 5 months ago
Labels: bug, better debugging, usability

#192 - Allow running tasks on active schema (Big DANGER mode disclaimer)

Issue - State: open - Opened by windiana42 5 months ago
Labels: enhancement, usability, speed

#191 - DuckDB: Fix Table rename with primary key/index

Issue - State: open - Opened by windiana42 5 months ago
Labels: bug

#190 - Move primary key generation for mssql backend after completely filling a table.

Pull Request - State: closed - Opened by windiana42 5 months ago

#189 - Implement bulk load to snowflake database (the driver has a mechanism to load parquet files via S3)

Issue - State: open - Opened by windiana42 5 months ago
Labels: usability, speed

#188 - Implement @input_stage_version decorator which provides inputs from two versions of the same stage before schema swapping

Pull Request - State: closed - Opened by windiana42 5 months ago

#187 - Better error message when executing subflow and input tasks not found in cache

Issue - State: open - Opened by windiana42 6 months ago
Labels: better debugging, usability, error messages

#186 - Implementation of snowflake backend.

Pull Request - State: closed - Opened by windiana42 6 months ago - 2 comments

#185 - Do not require connectorx for polars dematerialization

Pull Request - State: closed - Opened by nicolasmueller 6 months ago

#184 - Support Snowflake backend

Issue - State: closed - Opened by windiana42 6 months ago - 1 comment
Labels: enhancement

#183 - Watch duckdb 0.10.1 and sqlalchemy 2.0.29 compatibility issues

Issue - State: open - Opened by windiana42 6 months ago - 1 comment
Labels: documentation, dependencies, blocked external

#182 - Add group nodes for structuring flows, as task ordering barrier, and for clusting tasks in visualization

Pull Request - State: closed - Opened by windiana42 6 months ago - 5 comments

#181 - Add validation tasks to a Stage which get access to multiple schemas at once (last committed schema of this instance or another instance for this Stage)

Issue - State: closed - Opened by windiana42 6 months ago - 8 comments
Labels: enhancement, usability

#180 - Group tasks or stages in visualization

Issue - State: closed - Opened by windiana42 6 months ago - 5 comments
Labels: enhancement, usability

#179 - Implement imperative materialization.

Pull Request - State: closed - Opened by windiana42 6 months ago

#178 - Improve insert_into_in_query to treat comments in TSQL select statements

Issue - State: open - Opened by windiana42 6 months ago - 1 comment
Labels: good first issue, reliability, usability

#177 - fix(): Column called from triggers bug

Pull Request - State: closed - Opened by DominikZuercherQC 6 months ago

#176 - add autoincrement option

Pull Request - State: open - Opened by DominikZuercherQC 6 months ago

#175 - Provide a lot more examples for readthedocs.io documentation

Pull Request - State: closed - Opened by windiana42 6 months ago - 1 comment

#174 - Provide pydiverse.pipedag.version

Issue - State: open - Opened by windiana42 6 months ago
Labels: good first issue, usability

#173 - Move some or all combinatorial tests for cache_validation options to nightly tests.

Issue - State: open - Opened by windiana42 6 months ago
Labels: testing, speed

#172 - Add identity_insert materialisation option for MSSQL tablestores

Pull Request - State: open - Opened by DominikZuercherQC 6 months ago

#171 - Setting the isolation level for MSSQL table store to READ UNCOMMITED

Pull Request - State: closed - Opened by DominikZuercherQC 6 months ago

#170 - Propose redesign of cache validation options in configuration

Pull Request - State: closed - Opened by windiana42 6 months ago - 5 comments

#169 - mssql: Fast materialization/dematerialization based on bcpandas and similar bulk load techniques

Issue - State: open - Opened by windiana42 6 months ago
Labels: speed

#168 - Emergency Release 0.7.2: Implemented config for max_copy_operations, pool_size, pool_timeout.

Pull Request - State: closed - Opened by windiana42 6 months ago

#167 - Retry of producing a stage output

Issue - State: open - Opened by windiana42 7 months ago
Labels: enhancement, usability, refactoring

#166 - Add option to `Flow.run` to force execution of all tasks

Pull Request - State: closed - Opened by nicolasmueller 7 months ago - 5 comments

#165 - Disable cache validation for a specific run

Issue - State: closed - Opened by nicolasmueller 7 months ago
Labels: usability

#164 - Imperative Materialization

Issue - State: closed - Opened by windiana42 7 months ago - 2 comments
Labels: enhancement, usability, refactoring

#163 - Implement inline feature in pipedag.Table() to prevent materialization to database if not needed

Issue - State: open - Opened by windiana42 7 months ago - 2 comments
Labels: enhancement, usability, refactoring, speed

#162 - Fix #127 regarding UNLOGGED postgres tables during copy table.

Pull Request - State: closed - Opened by windiana42 7 months ago - 1 comment

#161 - Implement systematic retry logic especially for `sa.Table(autoload_with=)`

Issue - State: open - Opened by windiana42 7 months ago - 1 comment
Labels: partially done, testing, reliability, usability

#160 - Support nullable and non_nullable parameters in pipedag.Table() and refactor ddl/table-store/hook logic

Pull Request - State: closed - Opened by windiana42 7 months ago - 2 comments

#159 - Support setting nullable / non-nullable constraints in pipedag.Table()

Issue - State: closed - Opened by windiana42 7 months ago - 6 comments
Labels: enhancement, usability

#158 - Refactor DB2 dependency in generic code.

Pull Request - State: closed - Opened by windiana42 7 months ago - 2 comments

#157 - Fix duplicate table bug

Pull Request - State: closed - Opened by nicolasmueller 7 months ago

#156 - Disable kroki links by default.

Pull Request - State: closed - Opened by windiana42 7 months ago - 1 comment

#155 - Adding max_nchars_query_print option for MSSqlTableStore

Pull Request - State: closed - Opened by DominikZuercherQC 7 months ago - 2 comments

#154 - Smallish DDL extension

Pull Request - State: closed - Opened by windiana42 7 months ago

#153 - Fix numeric to float

Pull Request - State: closed - Opened by windiana42 7 months ago

#152 - Fix duckdb example documentation

Pull Request - State: closed - Opened by windiana42 7 months ago - 2 comments

#151 - Simplify logging inside tasks

Issue - State: open - Opened by windiana42 7 months ago
Labels: usability

#150 - Automatically add ExternalTableReference to auto_table list

Issue - State: open - Opened by windiana42 7 months ago - 1 comment
Labels: usability

#149 - Fix Polars and IBIS tests with DB2 URL and connectorX

Issue - State: open - Opened by windiana42 7 months ago - 1 comment
Labels: dependencies, reliability, usability

#148 - #142: Easier configuration setup

Pull Request - State: closed - Opened by windiana42 7 months ago - 3 comments

#147 - Add some unicode tests

Pull Request - State: closed - Opened by windiana42 7 months ago

#146 - Make max_length = 5000 configurable in mssql dialect for RawSQL Strings

Issue - State: closed - Opened by QuantCo-mr-T 7 months ago - 1 comment
Labels: good first issue, usability

#145 - Update link to table backends

Pull Request - State: closed - Opened by mohahf19 7 months ago

#144 - Fix task_context.input_tables (needed for DB2 input table locking)

Pull Request - State: closed - Opened by nicolasmueller 7 months ago

#143 - Documented input_type options and concepts for future enhancements.

Pull Request - State: closed - Opened by windiana42 7 months ago

#142 - Enable PipedagConfig construction from dictionary

Issue - State: closed - Opened by windiana42 7 months ago - 2 comments
Labels: usability

#141 - Check cache validity of lazy DataFrame tasks.

Issue - State: open - Opened by NicolasMuellerQC 7 months ago
Labels: good first issue, speed

#140 - Make lazy task output tables which do not provide a query string cache invalid for subsequent reading tasks

Pull Request - State: closed - Opened by nicolasmueller 7 months ago - 2 comments

#139 - Enable the creation of external TableReferences

Pull Request - State: closed - Opened by nicolasmueller 7 months ago - 6 comments

#138 - Fix `deep_map`

Pull Request - State: open - Opened by nicolasmueller 7 months ago - 6 comments

#137 - Copy task inputs to protect against modifications before execution time

Pull Request - State: open - Opened by nicolasmueller 8 months ago - 3 comments

#136 - Bump pre-commit/action from 3.0.0 to 3.0.1

Pull Request - State: closed - Opened by dependabot[bot] 8 months ago - 1 comment
Labels: dependencies

#135 - Use `polars-lts-cpu` instead of `polars` for x86_64 OS X

Pull Request - State: closed - Opened by nicolasmueller 8 months ago - 1 comment
Labels: bug, dependencies

#134 - make pytz and pytzdata optional dependencies

Pull Request - State: closed - Opened by windiana42 8 months ago

#133 - Make pytz optional dependency due to prefect

Pull Request - State: closed - Opened by windiana42 8 months ago - 1 comment

#132 - Reactivate date tests for duckdb

Issue - State: open - Opened by windiana42 8 months ago
Labels: testing, blocked external

#131 - Fix problems in tests after updating dependencies

Pull Request - State: closed - Opened by windiana42 9 months ago

#130 - Document SQLTableStore parameters in config.md

Pull Request - State: open - Opened by nicolasmueller 10 months ago

#129 - Bump actions/setup-python from 4 to 5

Pull Request - State: closed - Opened by dependabot[bot] 10 months ago - 1 comment
Labels: dependencies

#128 - Fix no materialization details

Pull Request - State: closed - Opened by nicolasmueller 10 months ago

#127 - Support for Postgres unlogged tables when copying tables

Issue - State: closed - Opened by nicolasmueller 10 months ago - 1 comment
Labels: bug

#126 - Support for Delta Loads and Table Partitioning

Issue - State: open - Opened by NicholasHoernleQC 10 months ago - 2 comments
Labels: enhancement

#125 - Postgres unlogged tables as a materialization detail

Pull Request - State: closed - Opened by nicolasmueller 10 months ago

#124 - DB2 RUNSTATS

Pull Request - State: closed - Opened by nicolasmueller 10 months ago

#123 - Instructions for getting pyodbc to run on OS X

Pull Request - State: closed - Opened by nicolasmueller 11 months ago

#122 - Materialization details

Pull Request - State: closed - Opened by nicolasmueller 11 months ago - 4 comments

GitHub / pydiverse/pydiverse.pipedag issues and pull requests