mozilla/gcp-ingestion issues and pull requests

#78 - Refactor Sink.Options to options.SinkOptions

Pull Request - State: closed - Opened by relud over 6 years ago - 2 comments

#78 - Refactor Sink.Options to options.SinkOptions

Pull Request - State: closed - Opened by relud over 6 years ago - 2 comments

#77 - Support optional doc id and auto-uuid generation for non-legacy doctypes

Issue - State: open - Opened by whd over 6 years ago - 2 comments
Labels: enhancement

#76 - Provide static identity() transform for clearer intent

Pull Request - State: closed - Opened by jklukas over 6 years ago

#75 - Remove empty lines between getters and setters in options

Pull Request - State: closed - Opened by jklukas over 6 years ago

#74 - Write tests for local IO in ingestion-beam's Sink pipeline

Issue - State: closed - Opened by relud over 6 years ago - 5 comments
Labels: test

#73 - Add MPL header to checkstyle for ingestion-beam

Issue - State: closed - Opened by relud over 6 years ago

#72 - Add a default processElement for input PubsubMessage

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#72 - Add a default processElement for input PubsubMessage

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#71 - Make mvn checkstyle:check work

Pull Request - State: closed - Opened by relud over 6 years ago

#70 - Add Deduplicate to Decoder

Pull Request - State: closed - Opened by relud over 6 years ago

#69 - Pull down mozilla-pipeline-schemas in pom.xml

Pull Request - State: closed - Opened by jklukas over 6 years ago

#68 - Use doctoc for generating markdown TOC

Pull Request - State: closed - Opened by jklukas over 6 years ago

#68 - Use doctoc for generating markdown TOC

Pull Request - State: closed - Opened by jklukas over 6 years ago

#67 - Move common pattern to MapElementsWithErrors

Pull Request - State: closed - Opened by relud over 6 years ago

#66 - Add ./bin/update-toc for generating a README table of contents

Pull Request - State: closed - Opened by jklukas over 6 years ago - 3 comments

#65 - Replace PubsubMessage with POJO in ingestion-beam

Issue - State: closed - Opened by relud over 6 years ago - 3 comments
Labels: enhancement

#64 - Generate TOC in README

Issue - State: closed - Opened by jklukas over 6 years ago

#63 - Fork google_checks.xml into our own checkstyle config

Pull Request - State: closed - Opened by jklukas over 6 years ago

#62 - Fork google_check.xml

Issue - State: closed - Opened by relud over 6 years ago

#61 - Use FileIO.writeDynamic to route output based on attributes

Pull Request - State: closed - Opened by jklukas over 6 years ago

#60 - Set default log level to WARNING

Pull Request - State: closed - Opened by relud over 6 years ago

#59 - Implement deduplication in decoder service

Issue - State: closed - Opened by relud over 6 years ago - 1 comment

#59 - Implement deduplication in decoder service

Issue - State: closed - Opened by relud over 6 years ago - 1 comment

#58 - Add metadata to payload in Decoder

Pull Request - State: closed - Opened by relud over 6 years ago

#58 - Add metadata to payload in Decoder

Pull Request - State: closed - Opened by relud over 6 years ago

#57 - Pre-load all schemas in ingestion-beam

Issue - State: closed - Opened by relud over 6 years ago

#57 - Pre-load all schemas in ingestion-beam

Issue - State: closed - Opened by relud over 6 years ago

#56 - Import mozilla-pipeline-schemas to ingestion-beam/src/main/resources

Issue - State: closed - Opened by relud over 6 years ago - 1 comment

#55 - Rename edge to ingestion-edge

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#54 - Add a test helper for running `main` and validating output on the filesystem

Issue - State: closed - Opened by jklukas over 6 years ago

#53 - Less verbose Println

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#52 - Implement initial schema validation

Pull Request - State: closed - Opened by relud over 6 years ago - 3 comments

#52 - Implement initial schema validation

Pull Request - State: closed - Opened by relud over 6 years ago - 3 comments

#51 - Move Decoder-specific transforms to separate package

Pull Request - State: closed - Opened by jklukas over 6 years ago

#51 - Move Decoder-specific transforms to separate package

Pull Request - State: closed - Opened by jklukas over 6 years ago

#50 - Restore ability to write to stdout

Pull Request - State: closed - Opened by jklukas over 6 years ago - 1 comment

#50 - Restore ability to write to stdout

Pull Request - State: closed - Opened by jklukas over 6 years ago - 1 comment

#49 - Fix stdout regression

Issue - State: closed - Opened by jklukas over 6 years ago

#49 - Fix stdout regression

Issue - State: closed - Opened by jklukas over 6 years ago

#48 - Refactor error logic

Pull Request - State: closed - Opened by jklukas over 6 years ago - 2 comments

#48 - Refactor error logic

Pull Request - State: closed - Opened by jklukas over 6 years ago - 2 comments

#47 - Consolidate error handling logic

Issue - State: closed - Opened by jklukas over 6 years ago - 1 comment

#47 - Consolidate error handling logic

Issue - State: closed - Opened by jklukas over 6 years ago - 1 comment

#46 - Add ParseUri to Decoder

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#46 - Add ParseUri to Decoder

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#45 - Rename Validation Service to Decoder

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#45 - Rename Validation Service to Decoder

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#44 - Prepare a talk for a technical review of ingestion

Issue - State: closed - Opened by relud over 6 years ago
Labels: enhancement

#43 - Add ParseUserAgent to Validation

Pull Request - State: closed - Opened by relud over 6 years ago - 3 comments

#42 - Refactor switch statements to methods on enums

Pull Request - State: closed - Opened by jklukas over 6 years ago

#41 - Implement Decompression in Validation Service

Pull Request - State: closed - Opened by relud over 6 years ago

#41 - Implement Decompression in Validation Service

Pull Request - State: closed - Opened by relud over 6 years ago

#40 - rename edge and prefix with ingestion-

Issue - State: closed - Opened by relud over 6 years ago - 4 comments

#40 - rename edge and prefix with ingestion-

Issue - State: closed - Opened by relud over 6 years ago - 4 comments

#39 - Rename Validation Service to Decoder

Issue - State: closed - Opened by relud over 6 years ago - 13 comments

#39 - Rename Validation Service to Decoder

Issue - State: closed - Opened by relud over 6 years ago - 13 comments

#38 - Add metrics to Sink and Validate

Issue - State: closed - Opened by jklukas over 6 years ago

#38 - Add metrics to Sink and Validate

Issue - State: closed - Opened by jklukas over 6 years ago

#37 - Implement GeoIP lookups for Validation Service

Pull Request - State: closed - Opened by relud over 6 years ago - 2 comments

#36 - Add multiple executions to checkstyle

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#35 - implement decoder service with Beam

Issue - State: closed - Opened by relud over 6 years ago

#34 - Rewrite Sink in Java

Pull Request - State: closed - Opened by relud over 6 years ago - 7 comments

#33 - Convert Sink from Scala to Java and document why

Issue - State: closed - Opened by relud over 6 years ago - 2 comments
Labels: enhancement

#32 - Add error output to sink

Pull Request - State: closed - Opened by relud over 6 years ago - 1 comment

#31 - Add BigQuery output to Sink

Pull Request - State: closed - Opened by relud over 6 years ago

#31 - Add BigQuery output to Sink

Pull Request - State: closed - Opened by relud over 6 years ago

#30 - WIP: Deserialize to ADTs

Pull Request - State: closed - Opened by jklukas over 6 years ago - 3 comments

#29 - Refactor string-typed pattern matches to ADTs

Pull Request - State: closed - Opened by jklukas over 6 years ago - 2 comments

#28 - Rename landfill to sink

Pull Request - State: closed - Opened by relud over 6 years ago

#28 - Rename landfill to sink

Pull Request - State: closed - Opened by relud over 6 years ago

#27 - Add text input/output file format to landfill

Pull Request - State: closed - Opened by relud over 6 years ago

#27 - Add text input/output file format to landfill

Pull Request - State: closed - Opened by relud over 6 years ago

#26 - Fix CircleCI badge

Pull Request - State: closed - Opened by jklukas over 6 years ago

#26 - Fix CircleCI badge

Pull Request - State: closed - Opened by jklukas over 6 years ago

#25 - Fix CircleCI badge

Pull Request - State: closed - Opened by jklukas over 6 years ago

#24 - Add options to landfill to read from and/or write to gcs in avro format

Issue - State: closed - Opened by relud over 6 years ago - 4 comments
Labels: enhancement

#23 - Add options to landfill optionally read and/or write only message payload

Issue - State: closed - Opened by relud over 6 years ago

#23 - Add options to landfill optionally read and/or write only message payload

Issue - State: closed - Opened by relud over 6 years ago

#22 - Refactor Sink to use FileIO.writeDynamic

Issue - State: closed - Opened by relud over 6 years ago

#22 - Refactor Sink to use FileIO.writeDynamic

Issue - State: closed - Opened by relud over 6 years ago

#21 - Update CI to use a smaller image for gcloud emulators

Pull Request - State: closed - Opened by relud over 6 years ago

#21 - Update CI to use a smaller image for gcloud emulators

Pull Request - State: closed - Opened by relud over 6 years ago

#20 - WIP: Very rough draft of TESTPLAN and LOADTEST thoughts

Pull Request - State: closed - Opened by pdehaan over 6 years ago - 1 comment

#20 - WIP: Very rough draft of TESTPLAN and LOADTEST thoughts

Pull Request - State: closed - Opened by pdehaan over 6 years ago - 1 comment

#19 - Implement Dataflow jobs with templates

Issue - State: closed - Opened by whd over 6 years ago - 1 comment

#18 - Add compression option to landfill

Issue - State: closed - Opened by relud over 6 years ago - 3 comments
Labels: enhancement

#17 - Replace Flask with Sanic in edge

Issue - State: closed - Opened by relud over 6 years ago - 5 comments
Labels: enhancement

#16 - Add static type checker

Issue - State: closed - Opened by jklukas over 6 years ago - 1 comment

#16 - Add static type checker

Issue - State: closed - Opened by jklukas over 6 years ago - 1 comment

#15 - Make edge tests use docker-compose

Issue - State: closed - Opened by relud over 6 years ago - 1 comment

#15 - Make edge tests use docker-compose

Issue - State: closed - Opened by relud over 6 years ago - 1 comment

#14 - Document how to configure the edge server

Issue - State: closed - Opened by relud over 6 years ago - 1 comment
Labels: bug

#14 - Document how to configure the edge server

Issue - State: closed - Opened by relud over 6 years ago - 1 comment
Labels: bug

#13 - Make edge server's metadata headers configurable

Issue - State: closed - Opened by relud over 6 years ago
Labels: enhancement, ingestion-edge

GitHub / mozilla/gcp-ingestion issues and pull requests