pydiverse/pydiverse.pipedag issues and pull requests

#121 - Drop DB2 nicknames

Pull Request - State: closed - Opened by nicolasmueller 11 months ago

#120 - Update PR template

Pull Request - State: closed - Opened by nicolasmueller 11 months ago

#120 - Update PR template

Pull Request - State: closed - Opened by nicolasmueller 11 months ago

#119 - DB2 development on OS X arm64

Pull Request - State: closed - Opened by nicolasmueller 11 months ago - 2 comments

#119 - DB2 development on OS X arm64

Pull Request - State: closed - Opened by nicolasmueller 11 months ago - 2 comments

#118 - [Fix] Cascade Schema dropping in IBM DB2

Issue - State: closed - Opened by MichalHeydelQC 11 months ago - 2 comments

#117 - Add a function to ad-hoc materialize a table for interactive debugging

Pull Request - State: closed - Opened by nicolasmueller 11 months ago

#117 - Add a function to ad-hoc materialize a table for interactive debugging

Pull Request - State: closed - Opened by nicolasmueller 11 months ago

#116 - Simplify debugging of materialization write-back

Issue - State: open - Opened by windiana42 11 months ago - 1 comment
Labels: partially done

#116 - Simplify debugging of materialization write-back

Issue - State: open - Opened by windiana42 11 months ago - 1 comment
Labels: partially done

#115 - Create "virtual" stacktrace which combines wiring time and task execution time

Issue - State: open - Opened by windiana42 about 1 year ago - 2 comments
Labels: enhancement, better debugging

#114 - pipedag-manage delete-schemas: support clearing unused schemas

Issue - State: open - Opened by windiana42 about 1 year ago
Labels: enhancement, usability

#114 - pipedag-manage delete-schemas: support clearing unused schemas

Issue - State: open - Opened by windiana42 about 1 year ago
Labels: enhancement, usability

#113 - for DB2 database: allow configuring table creation properties globally and/or per schema or table

Issue - State: closed - Opened by windiana42 about 1 year ago - 1 comment
Labels: usability, speed

#112 - for DB2 database: RUNSTAT for every table after it was filled and indexes created

Issue - State: closed - Opened by windiana42 about 1 year ago - 2 comments
Labels: reliability, usability, speed

#111 - Bump actions/checkout from 3 to 4

Pull Request - State: closed - Opened by dependabot[bot] about 1 year ago - 1 comment
Labels: dependencies

#110 - Apply database default case to table names during materialization

Issue - State: open - Opened by windiana42 about 1 year ago - 1 comment
Labels: usability

#110 - Apply database default case to table names during materialization

Issue - State: open - Opened by windiana42 about 1 year ago - 1 comment
Labels: usability

#109 - Disable warning that parquet table cache cannot dematerialize sa.Table

Issue - State: open - Opened by windiana42 about 1 year ago - 1 comment
Labels: good first issue, usability, error messages

#108 - Improve error message when polars frame is returned but connectorx is not installed

Issue - State: closed - Opened by windiana42 about 1 year ago

#108 - Improve error message when polars frame is returned but connectorx is not installed

Issue - State: closed - Opened by windiana42 about 1 year ago

#107 - Making `ignore_fresh_input` more explicit (or the docs)

Issue - State: closed - Opened by MarcAntoineSchmidtQC about 1 year ago - 1 comment

#106 - version 0.6.2 writes pandas tables with lowercase names to DB2

Issue - State: closed - Opened by windiana42 about 1 year ago - 2 comments

#106 - version 0.6.2 writes pandas tables with lowercase names to DB2

Issue - State: closed - Opened by windiana42 about 1 year ago - 2 comments

#105 - pipedag-manage command to list all instances

Issue - State: open - Opened by windiana42 about 1 year ago - 2 comments
Labels: usability

#105 - pipedag-manage command to list all instances

Issue - State: open - Opened by windiana42 about 1 year ago - 2 comments
Labels: usability

#104 - DB2 does not allow order_by CLOB column

Issue - State: closed - Opened by windiana42 about 1 year ago
Labels: bug

#104 - DB2 does not allow order_by CLOB column

Issue - State: closed - Opened by windiana42 about 1 year ago
Labels: bug

#103 - Implement Stage.tbls

Issue - State: open - Opened by windiana42 about 1 year ago - 2 comments
Labels: enhancement, usability

#102 - Optimize error messages for SQL errors

Issue - State: open - Opened by windiana42 over 1 year ago
Labels: usability, error messages

#102 - Optimize error messages for SQL errors

Issue - State: open - Opened by windiana42 over 1 year ago
Labels: usability, error messages

#101 - fix prefect 2 engine

Issue - State: closed - Opened by windiana42 over 1 year ago - 1 comment
Labels: bug

#101 - fix prefect 2 engine

Issue - State: closed - Opened by windiana42 over 1 year ago - 1 comment
Labels: bug

#100 - polars dataframe needs write_parquet not to_parquet

Issue - State: closed - Opened by windiana42 over 1 year ago - 2 comments
Labels: bug

#99 - support empty stage

Issue - State: closed - Opened by windiana42 over 1 year ago

#98 - make default string length in DB2 dataframe materialization configurable

Issue - State: open - Opened by windiana42 over 1 year ago
Labels: usability

#97 - fix type annotation errors in clear_metadata.py

Issue - State: closed - Opened by windiana42 over 1 year ago
Labels: bug

#96 - sqlalchemy does not have a type called Double

Issue - State: closed - Opened by QuantCo-mr-T over 1 year ago - 1 comment
Labels: bug

#95 - DatabaseLockManager requires schema creation permission

Issue - State: closed - Opened by QuantCo-mr-T over 1 year ago
Labels: bug

#94 - Get rid of `ibm_db_sa_fix_name` hack

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#93 - Refactor TableHooks for SQLTableStore + Support for pyarrow backed pandas DataFrames

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#92 - Reflect on global communication with tasks in multi-node setup

Issue - State: open - Opened by windiana42 over 1 year ago - 1 comment
Labels: testing, better debugging, usability

#91 - Optimize Log initialization for parallel execution and prefect integration

Issue - State: open - Opened by windiana42 over 1 year ago - 1 comment
Labels: enhancement, better debugging, usability

#90 - Add support for DuckDB

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#89 - Allow execution of a Subflow

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#88 - Feat: Implement DatabaseLockManager

Pull Request - State: closed - Opened by NMAC427 over 1 year ago
Labels: enhancement

#87 - Create helm chart to make it easier to setup Linux cloud VM for testing pipedag with docker

Issue - State: open - Opened by windiana42 over 1 year ago - 1 comment
Labels: better debugging

#86 - Fix tidypolars test.

Pull Request - State: closed - Opened by windiana42 over 1 year ago

#85 - Log exceptions produced by tasks, even if `fail_fast=False`

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#84 - Refactor deferred table store operations and properly handle renaming of tables

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#83 - Encrypt and authenticate IPC Server + Client communication

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#82 - Update public interface

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#81 - Restructure Dependencies (#80)

Pull Request - State: closed - Opened by NMAC427 over 1 year ago

#80 - Reorganize dependencies

Issue - State: closed - Opened by windiana42 over 1 year ago - 2 comments

#79 - consider creating a utils library in pydiverse

Issue - State: open - Opened by windiana42 over 1 year ago - 2 comments
Labels: refactoring

#78 - publish documentation per release version

Issue - State: closed - Opened by windiana42 over 1 year ago - 2 comments
Labels: documentation, enhancement

#77 - Implement local table cache that stores inputs/outputs of dataframe tasks in parquet

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#76 - Forward dataframes between tasks in parallel to SQL writeback

Issue - State: open - Opened by windiana42 over 1 year ago - 1 comment
Labels: enhancement, speed

#75 - Support <n> cache slots

Issue - State: open - Opened by windiana42 over 1 year ago - 1 comment
Labels: enhancement, refactoring, speed

#74 - Store categorical variables with automatic shadow table in relational DB

Issue - State: open - Opened by windiana42 over 1 year ago - 1 comment
Labels: enhancement

#73 - Local file caching for dataframe/eager tasks

Issue - State: closed - Opened by windiana42 over 1 year ago - 7 comments
Labels: enhancement

#72 - implemented ignore_task_version option

Pull Request - State: closed - Opened by windiana42 over 1 year ago

#71 - Add option that disables explicit version checking for some instances

Issue - State: closed - Opened by windiana42 over 1 year ago
Labels: enhancement

#70 - fix ibm_db_sa bug when copying dataframes from cache: uppercase table names by default

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 2 comments

#69 - Pandas <-> SQL type mapping and chunksize=100k

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#68 - Add polars and tidypolars support for pipedag

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#67 - Add support for IBIS as a way to programmatically create SQL (and tests)

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#66 - Use "select *" where possible instead of writing out all columns of a table

Issue - State: closed - Opened by windiana42 over 1 year ago
Labels: partially done

#65 - Increase tasks.output_json length

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#64 - Integration tests with various pandas and sqlalchemy versions

Issue - State: closed - Opened by windiana42 over 1 year ago - 1 comment
Labels: testing

#63 - Set isolation level for mssql and DB2

Issue - State: open - Opened by windiana42 over 1 year ago
Labels: reliability

#62 - Improve pessimistic lazy caching

Issue - State: open - Opened by windiana42 over 1 year ago - 5 comments
Labels: partially done

#61 - closes #57: replace format_exception with format_exc

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#60 - Temp fix CI by running DB2 tests separately with only one concurrent configuration

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#59 - Test _trigger_deferred_table_store_ops branch `started_ops_end = max(...)+1`

Issue - State: closed - Opened by windiana42 over 1 year ago - 1 comment
Labels: testing

#58 - Make sure multiple scope exceptions are transported through remote procedure call

Issue - State: closed - Opened by windiana42 over 1 year ago - 2 comments
Labels: better debugging

#57 - Really support python 3.9

Issue - State: closed - Opened by windiana42 over 1 year ago - 2 comments

#56 - Support pandas 2.0 + polars with apache arrow backed dataframes

Issue - State: closed - Opened by windiana42 over 1 year ago - 2 comments
Labels: enhancement, speed

#55 - New Feature: implement local file cache (i.e. parquet files) for tasks working on dataframes

Issue - State: closed - Opened by windiana42 over 1 year ago - 1 comment

#54 - Don't copy any data in case of 100% cache valid stages

Pull Request - State: closed - Opened by windiana42 over 1 year ago - 3 comments

#53 - Fail tests if they emit logger.error()

Issue - State: open - Opened by windiana42 over 1 year ago
Labels: better debugging

#52 - Limit context output for stack trace output of errors in IPCWorker/IPCServer scenario

Issue - State: open - Opened by windiana42 over 1 year ago - 1 comment
Labels: better debugging

#51 - Can we put an authentication token in communication between tasks and RunConfigServer?

Issue - State: closed - Opened by windiana42 over 1 year ago - 2 comments
Labels: security

#50 - Changing output table name should invalidate cache or should be accounted for in cache-copying

Issue - State: closed - Opened by windiana42 over 1 year ago
Labels: reliability

#49 - Common case fast: If all tasks in a stage are cache valid, then there should not be spent time to write data to a new schema

Issue - State: closed - Opened by windiana42 over 1 year ago

GitHub / pydiverse/pydiverse.pipedag issues and pull requests