HTTPArchive/data-pipeline issues and pull requests

#269 - In future trim down custom metrics from `payload`?

Issue - State: open - Opened by tunetheweb 15 days ago - 3 comments

#268 - Bump apache-beam[gcp] from 2.54.0 to 2.56.0

Pull Request - State: open - Opened by dependabot[bot] 2 months ago - 1 comment
Labels: dependencies, python

#267 - Bump black from 24.3.0 to 24.4.2

Pull Request - State: open - Opened by dependabot[bot] 2 months ago - 1 comment
Labels: dependencies, python

#266 - Bump black from 24.3.0 to 24.4.1

Pull Request - State: closed - Opened by dependabot[bot] 2 months ago - 2 comments
Labels: dependencies, python

#265 - Bump github/super-linter from 5.0.0 to 6

Pull Request - State: open - Opened by dependabot[bot] 2 months ago - 1 comment
Labels: dependencies, github_actions

#264 - Bump black from 24.3.0 to 24.4.0

Pull Request - State: closed - Opened by dependabot[bot] 3 months ago - 2 comments
Labels: dependencies, python

#263 - Consider clustering `all.requests` table by `page` or `rank`

Issue - State: open - Opened by tunetheweb 3 months ago - 1 comment

#262 - Split the custom metrics JSON into structured fields

Issue - State: open - Opened by rviscomi 3 months ago - 1 comment

#261 - Bump apache-beam[gcp] from 2.54.0 to 2.55.1

Pull Request - State: closed - Opened by dependabot[bot] 3 months ago - 2 comments
Labels: dependencies, python

#260 - Bump google-cloud-firestore from 2.14.0 to 2.16.0

Pull Request - State: open - Opened by dependabot[bot] 3 months ago - 1 comment
Labels: dependencies, python

#259 - Bump apache-beam[gcp] from 2.54.0 to 2.55.0

Pull Request - State: closed - Opened by dependabot[bot] 3 months ago - 2 comments
Labels: dependencies, python

#258 - Rewrite legacy pipeline

Pull Request - State: open - Opened by tunetheweb 3 months ago - 1 comment

#257 - Summary data in `all.requests` issues

Issue - State: closed - Opened by tunetheweb 3 months ago - 2 comments

#256 - Bump dataflow flex template build tag

Pull Request - State: open - Opened by github-actions[bot] 4 months ago

#255 - Bump black from 24.2.0 to 24.3.0

Pull Request - State: closed - Opened by dependabot[bot] 4 months ago - 1 comment
Labels: dependencies, python

#254 - Add Technologies Report workflow

Pull Request - State: open - Opened by giancarloaf 4 months ago - 1 comment

#253 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 4 months ago

#252 - Bump google-cloud-firestore from 2.14.0 to 2.15.0

Pull Request - State: closed - Opened by dependabot[bot] 4 months ago - 2 comments
Labels: dependencies, python

#251 - Tech report pipeline - add flex template

Pull Request - State: closed - Opened by giancarloaf 4 months ago - 1 comment

#250 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 4 months ago

#249 - Bump apache-beam[gcp] from 2.52.0 to 2.54.0

Pull Request - State: closed - Opened by giancarloaf 4 months ago - 1 comment

#248 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 4 months ago

#247 - More fixes to improve run

Pull Request - State: open - Opened by tunetheweb 4 months ago - 1 comment

#246 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 4 months ago

#245 - Dataflow job failure: crawl-combined-desktop-20240227-042336-753436z

Issue - State: open - Opened by rviscomi 4 months ago

#244 - Error -3 while decompressing data: incorrect header check

Issue - State: open - Opened by rviscomi 4 months ago - 1 comment

#243 - KeyError: 'wasm'

Issue - State: open - Opened by rviscomi 4 months ago - 3 comments

#242 - Bump apache-beam[gcp] from 2.52.0 to 2.54.0

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 1 comment
Labels: dependencies, python

#241 - Bump black from 23.12.1 to 24.2.0

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 1 comment
Labels: dependencies, python

#240 - Bump peter-evans/create-pull-request from 5 to 6

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 1 comment
Labels: dependencies, github_actions

#239 - Bump black from 23.12.1 to 24.1.1

Pull Request - State: closed - Opened by dependabot[bot] 5 months ago - 2 comments
Labels: dependencies, python

#238 - Tech report pipeline

Pull Request - State: closed - Opened by giancarloaf 5 months ago - 1 comment

#237 - Add monthly CWV Tech Report materialization query

Issue - State: open - Opened by rviscomi 5 months ago - 6 comments

#236 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 6 months ago - 2 comments

#235 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 6 months ago - 1 comment

#234 - Add wasm type

Pull Request - State: closed - Opened by tunetheweb 6 months ago - 1 comment

#233 - Bump apache-beam[gcp] from 2.52.0 to 2.53.0

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 2 comments
Labels: dependencies, python

#232 - Bump black from 23.11.0 to 23.12.1

Pull Request - State: closed - Opened by dependabot[bot] 6 months ago - 1 comment
Labels: dependencies, python

#231 - Bump actions/download-artifact from 3 to 4

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 2 comments
Labels: dependencies, github_actions

#230 - Bump actions/upload-artifact from 3 to 4

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 1 comment
Labels: dependencies, github_actions

#229 - Bump github/codeql-action from 2 to 3

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 1 comment
Labels: dependencies, github_actions

#228 - Bump black from 23.11.0 to 23.12.0

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 2 comments
Labels: dependencies, python

#227 - Bump google-github-actions/setup-gcloud from 1 to 2

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 1 comment
Labels: dependencies, github_actions

#226 - Bump actions/setup-python from 4 to 5

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 1 comment
Labels: dependencies, github_actions

#225 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 7 months ago

#224 - Bump google-github-actions/auth from 1 to 2

Pull Request - State: closed - Opened by dependabot[bot] 7 months ago - 1 comment
Labels: dependencies, github_actions

#223 - Bump apache-beam[gcp] from 2.43.0 to 2.52.0

Pull Request - State: closed - Opened by dependabot[bot] 8 months ago - 2 comments
Labels: dependencies, python

#222 - Investigate missing Top 1k home pages

Issue - State: open - Opened by rviscomi 8 months ago - 1 comment

#221 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 8 months ago - 1 comment

#220 - Fail build if tag is not set

Pull Request - State: closed - Opened by tunetheweb 8 months ago - 1 comment

#219 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 8 months ago

#218 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 8 months ago - 1 comment

#217 - Bump black from 23.7.0 to 23.11.0

Pull Request - State: closed - Opened by dependabot[bot] 8 months ago - 1 comment
Labels: dependencies, python

#216 - Add WASM type

Pull Request - State: closed - Opened by tunetheweb 8 months ago - 2 comments

#215 - Bump black from 23.7.0 to 23.10.1

Pull Request - State: closed - Opened by dependabot[bot] 9 months ago - 2 comments
Labels: dependencies, python

#214 - Bump black from 23.7.0 to 23.10.0

Pull Request - State: closed - Opened by dependabot[bot] 9 months ago - 2 comments
Labels: dependencies, python

#213 - Bump apache-beam[gcp] from 2.43.0 to 2.51.0

Pull Request - State: closed - Opened by dependabot[bot] 9 months ago - 2 comments
Labels: dependencies, python

#212 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 9 months ago

#211 - Add checks for empty Entry

Pull Request - State: closed - Opened by tunetheweb 9 months ago - 2 comments

#210 - Clean up `beam_bq_job_LOAD` partial tables

Issue - State: open - Opened by rviscomi 9 months ago - 1 comment

#209 - Extend workflow BigQuery check timeout from 30s to 5m

Pull Request - State: closed - Opened by rviscomi 9 months ago - 1 comment

#208 - Rerunning a workflow with existing BQ data doesn't skip its Dataflow job

Issue - State: closed - Opened by rviscomi 9 months ago - 1 comment

#207 - Bump black from 23.7.0 to 23.9.1

Pull Request - State: closed - Opened by dependabot[bot] 10 months ago - 2 comments
Labels: dependencies, python

#206 - Bump actions/checkout from 3 to 4

Pull Request - State: closed - Opened by dependabot[bot] 10 months ago - 1 comment
Labels: dependencies, github_actions

#205 - Bump apache-beam[gcp] from 2.43.0 to 2.50.0

Pull Request - State: closed - Opened by dependabot[bot] 10 months ago - 2 comments
Labels: dependencies, python

#204 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 12 months ago

#203 - Fix all pipeline to use FlatMapTuple properly

Pull Request - State: closed - Opened by tunetheweb 12 months ago - 1 comment

#202 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] 12 months ago

#201 - Fix cloud workflow build

Pull Request - State: closed - Opened by tunetheweb 12 months ago - 1 comment

#200 - Correct template build script

Pull Request - State: closed - Opened by tunetheweb 12 months ago - 1 comment

#199 - Estimates are not correct for clustered tables for non-admin users

Issue - State: open - Opened by tunetheweb 12 months ago - 1 comment

#198 - Bump apache-beam[gcp] from 2.43.0 to 2.49.0

Pull Request - State: closed - Opened by dependabot[bot] 12 months ago - 2 comments
Labels: dependencies, python

#197 - Bump black from 23.3.0 to 23.7.0

Pull Request - State: closed - Opened by dependabot[bot] 12 months ago - 1 comment
Labels: dependencies, python

#196 - Switch to FlatMapTuple for combined pipeline

Pull Request - State: closed - Opened by tunetheweb 12 months ago - 1 comment

#195 - Update README with more info

Pull Request - State: closed - Opened by tunetheweb 12 months ago - 1 comment

#194 - Consider dropping pages with cross-domain redirects

Issue - State: open - Opened by rviscomi about 1 year ago - 3 comments

#193 - More fixes to handle bad JSONs

Pull Request - State: closed - Opened by tunetheweb about 1 year ago - 2 comments

#192 - Bump apache-beam[gcp] from 2.43.0 to 2.48.0

Pull Request - State: closed - Opened by dependabot[bot] about 1 year ago - 8 comments
Labels: dependencies, python

#191 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] about 1 year ago

#190 - Handle bad JSONs

Pull Request - State: closed - Opened by tunetheweb about 1 year ago - 1 comment

#189 - Add `rank` field to `all.requests`

Issue - State: open - Opened by rviscomi about 1 year ago - 1 comment
Labels: enhancement

#188 - Incomplete HTML response bodies

Issue - State: open - Opened by rviscomi about 1 year ago - 1 comment
Labels: bug

#187 - Bump github/super-linter from 4.10.1 to 5.0.0

Pull Request - State: closed - Opened by dependabot[bot] about 1 year ago - 1 comment
Labels: dependencies, github_actions

#186 - Bump peter-evans/create-pull-request from 4 to 5

Pull Request - State: closed - Opened by dependabot[bot] about 1 year ago - 1 comment
Labels: dependencies, github_actions

#185 - Bump black from 22.12.0 to 23.3.0

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 1 comment
Labels: dependencies, python

#184 - Bump apache-beam[gcp] from 2.43.0 to 2.46.0

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 3 comments
Labels: dependencies, python

#183 - Bump actions/checkout from 2 to 3

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 1 comment
Labels: dependencies, github_actions

#182 - Bump apache-beam[gcp] from 2.43.0 to 2.45.0

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 2 comments
Labels: dependencies, python

#181 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] over 1 year ago - 1 comment

#180 - Update workflow to accept multiple crawl paths

Pull Request - State: closed - Opened by giancarloaf over 1 year ago - 1 comment

#179 - Add check for valid GZIPs

Pull Request - State: closed - Opened by giancarloaf over 1 year ago - 7 comments

#178 - Specify workflows service account in deploy GHA

Pull Request - State: closed - Opened by giancarloaf over 1 year ago - 1 comment

#177 - Add report generation to workflow (move off of worker VM)

Issue - State: open - Opened by giancarloaf over 1 year ago - 2 comments

#176 - Bump black from 22.12.0 to 23.1.0

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 2 comments
Labels: dependencies, python

#175 - Adding Firefox use counter data to HTTPArchive

Issue - State: open - Opened by zcorpan over 1 year ago - 8 comments

#174 - Add Google Container Registry to dependabot

Pull Request - State: closed - Opened by giancarloaf over 1 year ago - 1 comment

#173 - Bump dataflow flex template build tag

Pull Request - State: closed - Opened by github-actions[bot] over 1 year ago - 1 comment

#172 - Fix dataflow flex template version bumping github action

Pull Request - State: closed - Opened by giancarloaf over 1 year ago - 1 comment

#171 - Fix Google Cloud deployment GitHub Actions

Pull Request - State: closed - Opened by giancarloaf over 1 year ago - 1 comment

#170 - Bump github/super-linter from 4.10.0 to 4.10.1

Pull Request - State: closed - Opened by dependabot[bot] over 1 year ago - 1 comment
Labels: dependencies, github_actions

GitHub / HTTPArchive/data-pipeline issues and pull requests