GlobalFishingWatch/pipe-segment issues and pull requests

#189 - PIPELINE-2244 Added support to source messages where filter field is timestamp data type

Pull Request - State: closed - Opened by rdgfuentes about 2 months ago - 1 comment

#188 - PIPELINE-2125: Gruping by ssvid, then frag, then date, then bins.

Pull Request - State: open - Opened by smpiano 4 months ago - 1 comment

#187 - PIPELINE-2125: Gruping by ssvid, then frag, then date, then bins.

Pull Request - State: closed - Opened by smpiano 4 months ago

#186 - Release 4.4.0

Pull Request - State: closed - Opened by tomaslink 5 months ago - 1 comment

#185 - PIPELINE-2054: Improve testing | Part II: utils

Pull Request - State: open - Opened by tomaslink 5 months ago - 1 comment

#184 - Bump urllib3 from 2.2.1 to 2.2.2

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 2 comments
Labels: dependencies

#183 - PIPELINE-2001: Improve testing | Part I: transforms

Pull Request - State: open - Opened by tomaslink 6 months ago - 1 comment

#182 - PIPELINE-2036: Decouple CLI segment identity options from beam

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#181 - Release 4.3.0

Pull Request - State: closed - Opened by smpiano 6 months ago - 1 comment

#181 - Release 4.3.0

Pull Request - State: closed - Opened by smpiano 6 months ago - 1 comment

#180 - Sync.master.develop.merge

Pull Request - State: closed - Opened by smpiano 6 months ago - 1 comment

#180 - Sync.master.develop.merge

Pull Request - State: closed - Opened by smpiano 6 months ago - 1 comment

#179 - Sync.develop.master.rebase

Pull Request - State: closed - Opened by smpiano 6 months ago

#179 - Sync.develop.master.rebase

Pull Request - State: closed - Opened by smpiano 6 months ago

#178 - Merges the pipe-segment v4 and v3 versions

Pull Request - State: closed - Opened by smpiano 6 months ago

#178 - Merges the pipe-segment v4 and v3 versions

Pull Request - State: closed - Opened by smpiano 6 months ago

#177 - Fix git workflow doc

Pull Request - State: closed - Opened by tomaslink 6 months ago

#177 - Fix git workflow doc

Pull Request - State: closed - Opened by tomaslink 6 months ago

#176 - PIPELINE-2020: Decouple CLI segment options from beam

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#175 - PIPELINE-2013: Use the same docker image for launch and runtime environments

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#175 - PIPELINE-2013: Use the same docker image for launch and runtime environments

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#174 - PIPELINE-2008: Test code up to python 3.12

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#174 - PIPELINE-2008: Test code up to python 3.12

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#173 - PIPELINE-2020: Decouple cli segment options from beam

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#173 - PIPELINE-2020: Decouple cli segment options from beam

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#172 - Makes the segment step runs for the VMS data. Disabled satellite dependencies and fields not present.

Pull Request - State: closed - Opened by smpiano 6 months ago

#171 - PIPELINE-1908: Added support for partitioned tables as message source

Pull Request - State: closed - Opened by rdgfuentes 6 months ago - 1 comment

#171 - PIPELINE-1908: Added support for partitioned tables as message source

Pull Request - State: closed - Opened by rdgfuentes 6 months ago - 1 comment

#170 - PIPELINE-2013: Use only one docker image

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#170 - PIPELINE-2013: Use only one docker image

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#169 - Document git workflow

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#169 - Document git workflow

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#168 - Bump jinja2 from 3.0.2 to 3.1.4

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 1 comment
Labels: dependencies

#168 - Bump jinja2 from 3.0.2 to 3.1.4

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 1 comment
Labels: dependencies

#167 - Bump ujson from 1.35 to 5.4.0

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 1 comment
Labels: dependencies

#167 - Bump ujson from 1.35 to 5.4.0

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 1 comment
Labels: dependencies

#166 - PIPELINE-2008: Test code up to python 3.11

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#166 - PIPELINE-2008: Test code up to python 3.11

Pull Request - State: closed - Opened by tomaslink 6 months ago - 1 comment

#165 - Update README.md

Pull Request - State: closed - Opened by smpiano 6 months ago

#165 - Update README.md

Pull Request - State: closed - Opened by smpiano 6 months ago

#164 - PIPELINE-1967: Use pip-tools to compile dependencies.

Pull Request - State: closed - Opened by tomaslink 7 months ago - 1 comment

#164 - PIPELINE-1967: Use pip-tools to compile dependencies.

Pull Request - State: closed - Opened by tomaslink 7 months ago - 1 comment

#163 - PIPELINE-1875 Ensure to set all fields mode None as NULLABLE

Pull Request - State: closed - Opened by rdgfuentes 8 months ago - 1 comment

#163 - PIPELINE-1875 Ensure to set all fields mode None as NULLABLE

Pull Request - State: closed - Opened by rdgfuentes 8 months ago - 1 comment

#162 - PIPELINE-1875 Ensure destination tables are created even when there is no data

Pull Request - State: closed - Opened by rdgfuentes 8 months ago

#162 - PIPELINE-1875 Ensure destination tables are created even when there is no data

Pull Request - State: closed - Opened by rdgfuentes 8 months ago

#161 - PIPELINE-1908 Added support for partitioned tables as message source

Pull Request - State: closed - Opened by rdgfuentes 8 months ago - 1 comment

#161 - PIPELINE-1908 Added support for partitioned tables as message source

Pull Request - State: closed - Opened by rdgfuentes 8 months ago - 1 comment

#160 - PIPELINE-1855 Tweak pipe-segment to be used with vms sources

Pull Request - State: closed - Opened by rdgfuentes 8 months ago

#160 - PIPELINE-1855 Tweak pipe-segment to be used with vms sources

Pull Request - State: closed - Opened by rdgfuentes 8 months ago

#159 - Relocates the images of the project under new path

Pull Request - State: closed - Opened by smpiano 10 months ago

#159 - Relocates the images of the project under new path

Pull Request - State: closed - Opened by smpiano 10 months ago

#158 - Makes the segment step runs for the VMS data. Disabled satellite dependencies and fields not present.

Pull Request - State: closed - Opened by smpiano 12 months ago

#157 - Adds the seg_id as clustering field for the segment_info and disbles the PFE in the satellite_timing_offset

Pull Request - State: closed - Opened by smpiano about 1 year ago

#156 - Updates table and column descriptons and sets labels in segment_info.

Pull Request - State: closed - Opened by smpiano about 1 year ago

#155 - Adds missing configuration to vessel_info table

Pull Request - State: closed - Opened by smpiano over 1 year ago

#154 - Adds missing configuration to satellite_timing_offsets table

Pull Request - State: closed - Opened by smpiano over 1 year ago

#153 - Adds support for Python 3.8 and Apache Beam 2.49

Pull Request - State: closed - Opened by smpiano over 1 year ago

#152 - Pipe-v2.5: Adds support for python-3.8 and AB-2.49

Pull Request - State: closed - Opened by smpiano over 1 year ago

#151 - Jenn.qa cleanup

Pull Request - State: closed - Opened by jennvanosdel over 1 year ago

#150 - Whitelist the messages segmented strict to the schema before writing

Pull Request - State: closed - Opened by smpiano over 1 year ago

#149 - Removes the use of noise and shiptype fields deprecated by segment_identity_daily

Pull Request - State: closed - Opened by smpiano over 1 year ago

#148 - PIPELINE-1262: Fixes normalized and denormalized values missmatching on vessel info

Pull Request - State: closed - Opened by andres-arana over 1 year ago

#147 - Fixing error in reading fragments when want to re-runs segment with already present fragment_ tables.

Pull Request - State: closed - Opened by smpiano over 1 year ago

#146 - Switch the satellite offsets table from daily sharded to partitioned

Pull Request - State: closed - Opened by smpiano over 1 year ago - 1 comment

#145 - Jenn.qa cleanup

Pull Request - State: closed - Opened by jennvanosdel over 1 year ago

#144 - Adds support for labels to audit the jobs.

Pull Request - State: closed - Opened by smpiano over 1 year ago

#143 - split read fragments up by year to try to prevent a timeout

Pull Request - State: closed - Opened by smpiano about 2 years ago - 2 comments

#142 - Fix fragment compare binsperday increased

Pull Request - State: closed - Opened by smpiano about 2 years ago

#141 - Try to reduce shuffle

Pull Request - State: closed - Opened by bitsofbits about 2 years ago - 1 comment

#140 - Small segment_vessel_daily fix

Pull Request - State: closed - Opened by jennvanosdel about 2 years ago - 1 comment

#139 - Jenn.segment identity daily fixes

Pull Request - State: closed - Opened by jennvanosdel about 2 years ago

#138 - segment_identity_daily fixes

Pull Request - State: closed - Opened by jennvanosdel over 2 years ago - 2 comments

#137 - Add QA notebooks

Pull Request - State: closed - Opened by jennvanosdel over 2 years ago

#136 - Fix for issues with cumulative identities

Pull Request - State: closed - Opened by jennvanosdel over 2 years ago

#135 - Properly aggregate cumulative identities in segments

Pull Request - State: closed - Opened by jennvanosdel over 2 years ago - 1 comment

#134 - Fixing potential bugs related to discrepancies in messages_segmented and sat_time_offsets

Pull Request - State: closed - Opened by jennvanosdel over 2 years ago - 1 comment

#133 - Fixed write of the segment_identity_daily_ table when is in backfill mode

Pull Request - State: closed - Opened by smpiano over 2 years ago - 1 comment

#132 - sketch out how to write segment identity by day -- needed to add summ…

Pull Request - State: closed - Opened by bitsofbits over 2 years ago - 1 comment

#131 - Fixed write of the segment_identity_daily_ table when is in backfill mode

Pull Request - State: closed - Opened by smpiano over 2 years ago

#130 - Sat timing fixes part 2

Pull Request - State: closed - Opened by jennvanosdel over 2 years ago

#129 - Pipe-2.5: Increment AB version to 2.40 to maintain DF SDK support.

Pull Request - State: closed - Opened by smpiano over 2 years ago

#128 - Update the version of Apache beam to 2.40.0

Pull Request - State: closed - Opened by smpiano over 2 years ago

#127 - Fixes little issue, source table was not formatting from template and throw error

Pull Request - State: closed - Opened by smpiano over 2 years ago

#126 - Add Fragment Table

Pull Request - State: closed - Opened by bitsofbits over 2 years ago

#125 - Stabilize segments generation

Pull Request - State: closed - Opened by bitsofbits over 2 years ago

#124 - Fix add cumulative data

Pull Request - State: closed - Opened by bitsofbits over 2 years ago

#123 - Clean up deprecated code

Pull Request - State: closed - Opened by smpiano over 2 years ago

#122 - Feature/pipeline 735

Pull Request - State: closed - Opened by smpiano over 2 years ago

#121 - Fixes tests and Dataflow job posted and working well

Pull Request - State: closed - Opened by smpiano over 2 years ago - 2 comments

#120 - Fix hot keys and Add Atomic Identities

Pull Request - State: closed - Opened by bitsofbits almost 3 years ago - 1 comment

#119 - Segmenter won't run in yearly mode

Issue - State: open - Opened by bitsofbits over 3 years ago

#118 - Collect pipe-segment changes that are slated for first round of pipeline improvements

Pull Request - State: closed - Opened by bitsofbits over 3 years ago

#117 - Uses cloudbuild instead of Travis

Pull Request - State: closed - Opened by smpiano over 3 years ago

#116 - Adds support for Beam 2.28.0

Pull Request - State: closed - Opened by smpiano over 3 years ago

#115 - Remove points with likely bad timestamps due to satellite clock noise.

Pull Request - State: closed - Opened by bitsofbits almost 4 years ago - 1 comment

#114 - Uses the new source_date_range_nodash method

Pull Request - State: closed - Opened by andres-arana about 4 years ago

#113 - PIPELINE-155: Adds segment identity check

Pull Request - State: closed - Opened by andres-arana about 4 years ago

#112 - Adds segmenter_params to segment step

Pull Request - State: closed - Opened by smpiano about 4 years ago

#111 - Updates gpsdio-segment to 0.20.2 which supports back and forward time identities

Pull Request - State: closed - Opened by andres-arana about 4 years ago

GitHub / GlobalFishingWatch/pipe-segment issues and pull requests