conbench/conbench issues and pull requests

#1247 - c-bench: surface benchmark density "results per case"

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: UI/UX

#1246 - benchmark_result.py: do not round/truncate in input path (aggregates)

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: bug

#1245 - Controlled `float`->`str`: use numpy's Dragon4 implementation instead of sigfig.round (unleash the dragon 🐉)

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 2 comments

#1244 - c-bench tinyplot: show unit, link to last result

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 3 comments

#1243 - Conbench UI: adhere to "Data Visualization Standards"

Issue - State: open - Opened by jgehrcke over 2 years ago
Labels: UI/UX

#1242 - c-bench: "making the context clickable "

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: UI/UX

#1241 - c-bench per-case table: show time of most recently submitted result to filter out outdated data

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: UI/UX

#1240 - c-bench small multiples: link to "last benchmark result" from a tinyplot

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: UI/UX

#1239 - c-benchmarks: small multiples / tiny history plots: change axis labels

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1237 - benchmarks.py: fix accidental BMRT cache item insertion

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1236 - benchmarks.py: accidental key insert upon key lookup

Issue - State: closed - Opened by jgehrcke over 2 years ago - 2 comments
Labels: bug

#1235 - webapp container image: set up postgres-client-15, misc

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 2 comments

#1234 - Dockerfile: add postgresql-client for debugging

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1233 - UI tech overhaul with e.g. HTMX?

Issue - State: open - Opened by jgehrcke over 2 years ago

#1232 - benchadapt: "scrape commit info from local repo" feature, used by Arrow CI, was removed by #1135

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: bug

#1231 - Request for enhancement: Filter benchmark results by run reason and time window

Issue - State: closed - Opened by boshek over 2 years ago - 9 comments
Labels: api

#1230 - ui: certain numbers are missing across the webapp

Issue - State: open - Opened by austin3dickey over 2 years ago - 2 comments
Labels: bug, testing

#1229 - benchmark result compare page: rename to `/compare/benchmark-results/`

Issue - State: closed - Opened by austin3dickey over 2 years ago
Labels: UX - terminology

#1228 - benchmark result compare page: should display errors if present

Issue - State: closed - Opened by austin3dickey over 2 years ago - 1 comment

#1227 - Explore removing Run table (in favor of run_id on Benchmark_Result, also commit, hardware, etc)

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: DB schema, datamodel & caching & perf

#1226 - UI: "repo-less" runs show up prominently

Issue - State: closed - Opened by jgehrcke over 2 years ago - 3 comments
Labels: bug, UI/UX

#1225 - benchmark-centric view(s): increase BMRT cache size, display topN newest results with 'timeago'. various visual tweaks

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1224 - Various minor follow-ups: rename svs, reduce log verbosity,

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1223 - BMRT cache: use selectinload strategy

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 2 comments

#1221 - benchalerts: link to compare/benchmarks when we can

Pull Request - State: closed - Opened by austin3dickey over 2 years ago - 1 comment

#1220 - benchalerts: let GitHubCheckStep use the external_id attribute

Pull Request - State: closed - Opened by austin3dickey over 2 years ago

#1219 - BMRT cache: reduce cpu work in update

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1218 - BMRT cache: cpu work regression as of `sigfig.round()`

Issue - State: closed - Opened by jgehrcke over 2 years ago - 4 comments
Labels: datamodel & caching & perf

#1217 - BenchmarkResult: fix `.data` type, think through `.unit` type, use more `.measurements`

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 2 comments

#1216 - kube-prometheus: bump retention time by factor 2

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 2 comments

#1215 - c-benchmarks landing page: case-insensitive sort order, small tweaks

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1214 - BenchmarkResult: fix SEM calculation (division by zero)

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1213 - BMRT cache: process database response in streaming fashion (reduce peak mem consumption during DB query)

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 4 comments

#1212 - bug: RetryingHTTPClient timeout 2-tuple mixup

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: bug

#1211 - BMRT cache: consumes too much memory

Issue - State: closed - Opened by jgehrcke over 2 years ago - 3 comments
Labels: datamodel & caching & perf

#1210 - api.compare.runs: don't force repos to match

Pull Request - State: closed - Opened by austin3dickey over 2 years ago - 1 comment

#1209 - History plot on the compare benchmark result page?

Issue - State: closed - Opened by jonkeane over 2 years ago - 4 comments
Labels: UI/UX

#1208 - retrying http client: better log msg for 5xx etc (add word "retryable")

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1207 - c-benchmarks: work on 'landing page' a bit

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1206 - benchclients: fix oversight, invert connect() and recv() timeout

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#1205 - BMRT cache: use slots-based minimal typed objects instead of massive and magic SQLAlchemy-flavored objects, misc

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 4 comments

#1204 - in testing, use jinja2.StrictUndefined

Issue - State: open - Opened by jgehrcke over 2 years ago
Labels: devprod, testing

#1203 - app run page: fix page when run doesn't have a commit

Pull Request - State: closed - Opened by austin3dickey over 2 years ago

#1202 - benchalerts: update to latest API changes

Pull Request - State: closed - Opened by austin3dickey over 2 years ago - 2 comments

#1201 - UI: run view: /app/results.py", line 199, in _display_time 'NoneType' object is not subscriptable

Issue - State: closed - Opened by jgehrcke over 2 years ago
Labels: bug

#1200 - landing page: 'None' has no attribute 'hash' / run.commit can be None

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: bug

#1199 - landing page: handle run.commit being `None`, tweak err handler, bump djlint

Pull Request - State: closed - Opened by jgehrcke over 2 years ago

#1197 - api/run: does not work for a run without a commit

Issue - State: closed - Opened by austin3dickey over 2 years ago
Labels: bug, api

#1196 - benchalerts: handle run comparisons without commits

Issue - State: closed - Opened by austin3dickey over 2 years ago - 1 comment
Labels: benchalerts

#1195 - webapp: consolidate exception treatment in HTTP handlers

Pull Request - State: closed - Opened by jgehrcke over 2 years ago

#1193 - RFC: a bit of UI work on the landing page

Issue - State: open - Opened by jgehrcke over 2 years ago - 2 comments
Labels: UI/UX

#1191 - benchalerts: notify/fail if there are no results on the contender

Issue - State: closed - Opened by austin3dickey over 2 years ago - 3 comments
Labels: benchalerts

#1189 - benchalerts: Use the comparison URL instead of entity URL (or in addition to)?

Issue - State: closed - Opened by jonkeane over 2 years ago - 1 comment
Labels: benchalerts

#1186 - benchalerts: use new ConbenchClient

Issue - State: closed - Opened by austin3dickey over 2 years ago - 1 comment
Labels: benchalerts

#1181 - webapp exception handler fails with `...Error is not JSON serializable`

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: devprod

#1175 - run page: Surface new comparisons in UI

Issue - State: closed - Opened by austin3dickey over 2 years ago - 1 comment
Labels: UI/UX

#1172 - guarantee BenchmarkResult.mean again, add clarity in many ways, introduce 'single value summary' to feed plot and history api

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 3 comments

#1169 - regression: BenchmarkResult.mean can be None for non-errored results, history analysis assumes numeric value

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: bug

#1165 - API: allow submitting runs with known repository but unknown commit hash

Issue - State: open - Opened by austin3dickey over 2 years ago - 11 comments

#1159 - ci instability: Service "grafana" is invalid: spec.ports[0].nodePort: port is already allocated

Issue - State: closed - Opened by jgehrcke over 2 years ago - 2 comments
Labels: ci-instability

#1155 - benchmark result view: hist plot render: sigfig: `ValueError: invalid input Character "n" (position 1, state A)`

Issue - State: closed - Opened by jgehrcke over 2 years ago - 4 comments
Labels: bug

#1154 - benchmark-centric UI/UX: ideas, initial observations and feedback

Issue - State: open - Opened by jgehrcke over 2 years ago - 2 comments
Labels: UI/UX

#1139 - DB schema: do not use user-given data in primary key (insert order invariant)

Issue - State: closed - Opened by jgehrcke over 2 years ago - 5 comments
Labels: DB schema

#1136 - API: require non-short commit hashes

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 2 comments

#1135 - benchadapt: consolidate the "github-flavored commit info" acquisition and communication

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 4 comments

#1134 - Tighten API spec with respect to GitHub commit information, adjust tests

Pull Request - State: closed - Opened by jgehrcke over 2 years ago - 8 comments

#1117 - instability: ci / db-migrations: `psycopg2.OperationalError) could not connect to server: Connection refused`

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: ci-instability

#1115 - regression detection: method comparing two (multisample) results

Issue - State: open - Opened by jgehrcke over 2 years ago - 1 comment
Labels: measurement method & analysis

#1113 - ui: c-benchmarks/<bname>: show 'last result' column, fix TypeError for partial multisample results

Pull Request - State: closed - Opened by jgehrcke over 2 years ago

#1109 - deploy HTTP reverse proxy between ALB and API containers

Issue - State: open - Opened by jgehrcke over 2 years ago
Labels: api latency, k8s-and-infra

#1108 - client libraries: HTTP API abstraction: take inspiration from others: beautiful API wrapper (robust, great errors, ...)

Issue - State: open - Opened by jgehrcke over 2 years ago - 1 comment

#1100 - benchmark result submission: do not require time_unit?

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: api JSON schemas and spec

#1078 - api/run: return more candidate baseline runs

Pull Request - State: closed - Opened by austin3dickey over 2 years ago - 7 comments

#1051 - z-scores: Simplify distribution query

Issue - State: open - Opened by austin3dickey over 2 years ago - 1 comment
Labels: api latency, method: lookback z-score, maintainability, datamodel & caching & perf

#1035 - pragmatic profiling: find out what it takes to fetch many BenchmarkResult objects for building an HTTP response

Issue - State: open - Opened by jgehrcke over 2 years ago - 4 comments
Labels: datamodel & caching & perf

#999 - /api endpoints: JSON serialization may dominate processing time, use orjson

Issue - State: open - Opened by jgehrcke over 2 years ago - 1 comment
Labels: api latency

#998 - local dev: segfault in gunicorn worker process

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment

#986 - benchalerts: first-class use of the `@github-actions` app token

Issue - State: open - Opened by austin3dickey over 2 years ago
Labels: benchalerts

#985 - benchalerts: ability to edit previous comment

Issue - State: open - Opened by austin3dickey over 2 years ago - 1 comment
Labels: benchalerts

#979 - benchalerts: error handlers should apply to steps, not a whole pipeline

Issue - State: open - Opened by austin3dickey over 2 years ago
Labels: benchalerts

#976 - document a method for how to import a sql dump into the docker-compose-managed db

Issue - State: open - Opened by jgehrcke over 2 years ago - 7 comments
Labels: documentation, DB schema

#975 - proposal: remove multi-run support in GET /benchmarks/?run_id=

Issue - State: closed - Opened by jgehrcke over 2 years ago - 3 comments
Labels: api, api latency

#969 - benchalerts: report how many commits ago was the last matching run on

Issue - State: open - Opened by austin3dickey over 2 years ago - 3 comments
Labels: benchalerts

#968 - webapp: skip API layer re-serialization of benchmark results

Issue - State: open - Opened by jgehrcke over 2 years ago - 1 comment
Labels: maintainability

#967 - tests: cover UI /benchmark-results/<benchmark-result-id>

Issue - State: closed - Opened by jgehrcke over 2 years ago - 2 comments
Labels: maintainability, testing

#959 - towards systematic method testing: make it easy to download timeseries for scenario lib

Issue - State: open - Opened by jgehrcke over 2 years ago - 1 comment
Labels: devprod, UI/UX

#955 - api/compare: distinguish the "best" benchmark for a given case/context/run

Issue - State: open - Opened by austin3dickey over 2 years ago - 7 comments

#945 - ci instability: ci/libraries: Test benchalerts: test_alert_pipeline[app] fails with 422 err

Issue - State: closed - Opened by jgehrcke over 2 years ago - 4 comments
Labels: ci-instability, benchalerts, testing

#929 - Explore using PostgreSQL or ClickHouse as "timeseries database"?

Issue - State: open - Opened by jgehrcke over 2 years ago - 7 comments
Labels: api latency, DB schema, datamodel & caching & perf

#904 - commit metadata: if acquisition failed once, it can be retried later (DB scan / consistency)

Issue - State: open - Opened by jgehrcke over 2 years ago - 3 comments
Labels: UX - robustness, git repository awareness

#895 - ci instability: Exception: Hourly GitHub HTTP API quota exhausted

Issue - State: open - Opened by jgehrcke over 2 years ago - 11 comments
Labels: ci-instability

#881 - ci instability: error: timed out waiting for the condition on pods/conbench-deployment ...

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: ci-instability

#872 - UI & API endpoints for collections: build time range filters / "The Time Window Paradigm"

Issue - State: open - Opened by jgehrcke over 2 years ago - 2 comments
Labels: UI/UX, api, api JSON schemas and spec

#871 - API endpoints: allow for reading data in chunks (introduce pagination)

Issue - State: closed - Opened by jgehrcke over 2 years ago - 4 comments
Labels: api, api JSON schemas and spec

#864 - discuss: consistency of Commit between BenchmarkResult and Run (and other properties)

Issue - State: closed - Opened by jgehrcke over 2 years ago - 4 comments
Labels: api JSON schemas and spec

#863 - review and resolve the challenging topic of implicit Run creation

Issue - State: closed - Opened by jgehrcke over 2 years ago - 1 comment
Labels: api JSON schemas and spec

#862 - timeseries fingerprint: use this to define comparability (re-think term "tags" and "case" -- generic label set, with fingerprint being hash thereof)

Issue - State: closed - Opened by jgehrcke over 2 years ago - 14 comments
Labels: documentation, UX - terminology

GitHub / conbench/conbench issues and pull requests