mbta/data_platform issues and pull requests

#122 - [elixir] chore: new ODS tables to ingest

Pull Request - State: closed - Opened by rymarczy about 1 month ago - 1 comment

#121 - [elixir] chore: new EDW.DEVICE_LAST_STATE table to ingest

Pull Request - State: closed - Opened by grejdi-mbta 2 months ago

#120 - [elixir] feat: add member dimension table for processing

Pull Request - State: closed - Opened by grejdi-mbta 4 months ago - 1 comment

#119 - [elixir] fix: update kpi monthly sldc table to active

Pull Request - State: closed - Opened by grejdi-mbta 7 months ago - 3 comments

#118 - [elixir] fix: dmap ingestion implementation is not complete, so turn off

Pull Request - State: closed - Opened by grejdi-mbta 7 months ago - 1 comment

#117 - [elixir] feat: add kpi monthly sldc table to be ingested

Pull Request - State: closed - Opened by grejdi-mbta 7 months ago - 1 comment

#116 - [elixir] feat: add admin app

Pull Request - State: open - Opened by grejdi-mbta 10 months ago - 2 comments

#115 - [elixir] feat: add apikey to the header for dmap fetch

Pull Request - State: closed - Opened by grejdi-mbta 10 months ago - 1 comment

#114 - [elixir] fix: athena error message when partition exists

Pull Request - State: closed - Opened by grejdi-mbta 11 months ago - 2 comments

#113 - [elixir] feat: add is_active to cubic tables and filter not active from ingestion

Pull Request - State: closed - Opened by grejdi-mbta 11 months ago - 2 comments

#112 - [elixir] chore: update elixir mix versions

Pull Request - State: open - Opened by grejdi-mbta 11 months ago

#111 - [elixir] fix: schedule dmap tests failing.

Pull Request - State: closed - Opened by grejdi-mbta 11 months ago - 12 comments

#110 - [elixir] feat: add athena partition for each load

Pull Request - State: closed - Opened by grejdi-mbta 11 months ago - 5 comments

#109 - [elixir] fix: aws cert update

Pull Request - State: closed - Opened by grejdi-mbta 12 months ago - 1 comment

#108 - [elixir] feat: get response body on dmap fetch error

Pull Request - State: closed - Opened by grejdi-mbta 12 months ago - 1 comment

#107 - [elixir] feat: hide any url details if dmap errors on fetch

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 1 comment

#106 - [elixir] hotfix: disable dmap until we figure out issue in ingestion

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 2 comments

#105 - [elixir] fix: updates dmap paths and use api keys

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 2 comments

#104 - [elixir] fix: disable load partitions for now

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 1 comment

#103 - [elixir] feat: only run athena queries within a workgroup

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 1 comment

#102 - [elixir] feat: scheduled loading of partitions in athena

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 5 comments

#101 - feat: use AWS IAM role instead of IAM user

Pull Request - State: closed - Opened by ianwestcott about 1 year ago - 3 comments

#100 - [env+ga] fix: fix versions for building local and on github

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 2 comments

#99 - [gh] fix: enable deploy actions for live rollout

Pull Request - State: closed - Opened by grejdi-mbta about 1 year ago - 2 comments

#98 - chore: update AWS cert check

Pull Request - State: closed - Opened by napilla over 1 year ago - 1 comment

#97 - [elixir] feat: check DMAP config values before running DMAP fetch process

Pull Request - State: closed - Opened by napilla over 1 year ago - 3 comments

#96 - [elixir] feat: add language table for ingestion

Pull Request - State: closed - Opened by napilla over 1 year ago - 1 comment

#95 - [elixir] fix: ingestion process crashes when missing schema file

Pull Request - State: closed - Opened by grejdi-mbta almost 2 years ago - 2 comments

#94 - [data] fix: add PK info for dfm sample data

Pull Request - State: closed - Opened by grejdi-mbta almost 2 years ago - 2 comments

#93 - [elixir] feat: add journal entry table for ingestion

Pull Request - State: closed - Opened by grejdi-mbta almost 2 years ago - 1 comment

#92 - [elixir] feat: add athena partition after successful glue job

Pull Request - State: closed - Opened by grejdi-mbta almost 2 years ago

#91 - [elixir] fix: checksum for aws rds cert

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago

#90 - [elixir] feat: rearrange genserver logic

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 9 comments

#89 - [elixir] fix: checksum for aws rds cert

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago

#88 - [elixir] fix: ehmon config in the correct place

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 1 comment

#87 - [elixir] hotfix: get only one record for the ods load snapshot

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 2 comments

#86 - [elixir] feat: oban will wait for jobs to finish before shutting down

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 3 comments

#85 - [elixir] feat: check schema prior to starting ingestion

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 4 comments

#84 - [elixir] fix: inserting of ready loads should cover loads from the last 24 hours

Pull Request - State: open - Opened by grejdi-mbta about 2 years ago

#83 - [elixir] fix: processing of ODS loads without snapshot load

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 7 comments

#82 - [elixir] feat: add info for args when worker has exception

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 2 comments

#81 - [db] feat: more ODS tables for ingestion

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 1 comment

#80 - [db] feat: add agg_daily_fareprod dmap feed

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 1 comment

#79 - [python] fix: pylint warnings and python version set

Pull Request - State: closed - Opened by grejdi-mbta about 2 years ago - 1 comment

#78 - [elixir] limit query for ready_for_archiving/erroring loads

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#77 - [elixir] feat: insert new objects only if new since last inserted load

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#76 - [db] feat: add new dmap agg_daily_fareprod table for consumption

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#75 - [elixir] feat: limit how many ready loads to start ingestion for

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 10 comments

#74 - [elixir] feat: move DFM files out of incoming

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 15 comments

#73 - [elixir] feat: adjust to querying loads with only recent s3 objects

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 8 comments

#72 - [elixir] fix: start transaction when something will be inserted

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#71 - [elixir] feat: adjust timers and oban async queue

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#70 - [elixir] feat: add ehmon for monitoring erlang stats

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#69 - [elixir] re-add chunking by size and number

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#68 - [db] add indexes for load tables

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#67 - [elixir] feat: set pool size for dev.

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 4 comments

#66 - [db] add all known DMAP feeds

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#65 - [elixir+python] add raw for unprocessed springboard loads

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 9 comments

#64 - [python+elixir] type casting for springboard

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#63 - [elixir] feat: add fetch dmap release task so we can refetch dataset

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#62 - [elixir] feat: make archive/error jobs more error proof

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 4 comments

#61 - [elixir] feat: release tasks to retry archive/error jobs that have failed

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 4 comments

#60 - [doc] feat: update local setup READMEs

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#59 - [elixir] feat: make schedule dmap task a release task

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#58 - [elixir] fix: make dmap envs more specific

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#57 - [elixir] bug: allow empty datasets to be passed into dmap feed update

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#56 - [elixir] feat: ability to execute a schedule DMAP fetching upon running the task.

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 7 comments

#55 - [doc] feat: ADR for standardizing all Cubic data to string type

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 9 comments

#54 - [elixir] fix: further filter objects coming in for ingestion

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#53 - [elixir+db] feat: add EDW.SVN_INCIDENT ODS table rec for ingestion.

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#52 - [elixir] feat: schedule dmap worker

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#51 - [elixir] feat: use stream for downloading and uploading

Pull Request - State: open - Opened by grejdi-mbta over 2 years ago

#50 - [elixir] fix: checksum for aws rds cert

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#49 - [elixir] feat: fetch dmap worker

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 6 comments

#48 - [db] feat: add more ODS table records

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#47 - [doc] feat: ADR for implementation of fetching DMAP data

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 4 comments

#46 - [elixir] fix: glue throttling because of aggressiveness on oban's part.

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 12 comments

#45 - [elixir+python] feat: refactor for generic ingestion and ODS ingestion

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 6 comments

#44 - [elixir+python] feat: refactor for generic use of ingestion

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 4 comments

#43 - [ex_cubic_ods_ingestion] feat: add rest of ODS tables

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 4 comments

#42 - [ex_cubic_ods_ingestion] feat: chunk loads by max num and size

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#41 - [py_cubic_ods_ingestion] feat: add pyspark testing + glue containers

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#40 - [code] fix: adjust where the cert is placed before release

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#39 - [code] fix: rds has ssl warning upon connection

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#38 - [code] feat: clean up repo and remove older python work

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#37 - [doc] feat: ADR for refactoring ingestion to account for DMAP/Kafka data

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 5 comments

#36 - [code] fix: broken testing due to wrong config setup

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 5 comments

#35 - [code] fix: handle retries on ingest worker error

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 5 comments

#34 - feat: add terraform area for local development resources

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#33 - [code] fix: wrong table is referenced for change tracking load

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#32 - [db] feat: add use_transaction table record to cubic_ods_tables

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#31 - [code] fix: rename glue job to conform to ECS config update

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#30 - [code] fix: snapshot now in iso 8601 + fix test

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 1 comment

#29 - [code] fix: config adjust to specify test db. add python package to glue job.

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#28 - [code] fix: add oban logger + rename configs

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#27 - [db] add sample table as record in cubic_ods_tables

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 4 comments

#26 - [code] feat: add linting with pylint

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 2 comments

#25 - [code] fix: tests were not working with prefixes env variables set

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 3 comments

#24 - [infra] feat: add deploy workflows

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago - 14 comments

#23 - [infra] feat: add workflows for elixir/glue deploys

Pull Request - State: closed - Opened by grejdi-mbta over 2 years ago

GitHub / mbta/data_platform issues and pull requests