Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / eleutherai/elk issues and pull requests

#299 - Roadmaps of elk

Issue - State: open - Opened by merlinarer about 2 months ago

#298 - Can this code suppot free-form QA task?

Issue - State: open - Opened by Tan-Hexiang 5 months ago - 2 comments

#297 - pip install from git doesn't work

Issue - State: open - Opened by Butanium 10 months ago

#296 - Made into a `pytest.fixture`

Pull Request - State: open - Opened by RohitRathore1 10 months ago - 2 comments

#294 - inference server duplication check

Pull Request - State: closed - Opened by AlexTMallen about 1 year ago

#293 - Answer tokenization

Pull Request - State: closed - Opened by AlexTMallen about 1 year ago

#292 - MAJOR UPDATE: Remove CCS, VINC

Pull Request - State: closed - Opened by AlexTMallen about 1 year ago

#291 - Add log softmax to model_logits

Pull Request - State: closed - Opened by norabelrose about 1 year ago

#290 - "None" ensembling for classfication accuracy

Pull Request - State: closed - Opened by derpyplops about 1 year ago - 1 comment

#289 - Add option to save probabilities

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago - 1 comment

#288 - add platt scaling for burns + fix for leace

Pull Request - State: closed - Opened by lauritowal over 1 year ago

#287 - Add `--no_balance` flag to not balance datasets

Pull Request - State: open - Opened by AlexTMallen over 1 year ago

#286 - Select metric choice for visualization

Pull Request - State: open - Opened by reaganjlee over 1 year ago

#285 - [Don't merge this] Platt burns

Pull Request - State: closed - Opened by derpyplops over 1 year ago

#284 - fix: remove first layer (embeddings) from layer_indices

Pull Request - State: closed - Opened by lauritowal over 1 year ago

#283 - remove calling self.leace.update twice

Pull Request - State: closed - Opened by lauritowal over 1 year ago

#282 - my fixes for layer ensembling

Pull Request - State: closed - Opened by derpyplops over 1 year ago
Labels: Draft

#281 - Update README.md

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago

#280 - Fix transfer plot bug

Pull Request - State: closed - Opened by reaganjlee over 1 year ago - 1 comment

#279 - Fix sweep ds config bug

Pull Request - State: closed - Opened by reaganjlee over 1 year ago

#278 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: open - Opened by pre-commit-ci[bot] over 1 year ago

#277 - output LM stats for , fix debug logging

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago

#276 - Evaluate LM output performance in `elk eval`

Issue - State: closed - Opened by norabelrose over 1 year ago
Labels: enhancement

#275 - Fix bug with non-editable installs

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#274 - remove duplicate normalization

Pull Request - State: closed - Opened by lauritowal over 1 year ago

#273 - ignore embeddings in extraction process

Pull Request - State: closed - Opened by lauritowal over 1 year ago

#272 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: closed - Opened by pre-commit-ci[bot] over 1 year ago

#271 - train probe per prompt

Pull Request - State: open - Opened by derpyplops over 1 year ago

#270 - Fix seed not exposed

Pull Request - State: closed - Opened by AugustasMacijauskas over 1 year ago

#269 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: closed - Opened by pre-commit-ci[bot] over 1 year ago

#268 - Linear Discriminant Analysis MVP

Pull Request - State: open - Opened by norabelrose over 1 year ago

#267 - Remove check_separability

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#266 - fix argument passthrough for sweep

Pull Request - State: open - Opened by derpyplops over 1 year ago

#265 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: closed - Opened by pre-commit-ci[bot] over 1 year ago

#263 - Fixes 256 - ValueError mutable default

Pull Request - State: closed - Opened by artkpv over 1 year ago - 2 comments

#261 - Use shrinkage for (cross-)covariance estimation

Issue - State: open - Opened by norabelrose over 1 year ago - 3 comments

#260 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: closed - Opened by pre-commit-ci[bot] over 1 year ago

#259 - Ensembling over layers

Pull Request - State: open - Opened by lauritowal over 1 year ago - 1 comment

#258 - Add per prompt norm

Pull Request - State: closed - Opened by lauritowal over 1 year ago

#257 - --max_examples seems to be broken

Issue - State: closed - Opened by derpyplops over 1 year ago - 2 comments

#256 - Cannot run README command `elk elicit microsoft/deberta-v2-xxlarge-mnli imdb`

Issue - State: open - Opened by rusheb over 1 year ago - 2 comments

#255 - Write smoke test for eval

Issue - State: open - Opened by derpyplops over 1 year ago
Labels: enhancement, good first issue

#254 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: closed - Opened by pre-commit-ci[bot] over 1 year ago

#253 - Add git hash in config

Pull Request - State: closed - Opened by derpyplops over 1 year ago

#252 - Use concept-erasure implementation of LEACE and SAL

Pull Request - State: closed - Opened by norabelrose over 1 year ago - 2 comments

#251 - Fix reading datasets with slash

Pull Request - State: closed - Opened by derpyplops over 1 year ago

#250 - Add check for model and dataset existence before running

Pull Request - State: closed - Opened by derpyplops over 1 year ago

#249 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: closed - Opened by pre-commit-ci[bot] over 1 year ago

#248 - Visualizations (with more comments)

Pull Request - State: closed - Opened by derpyplops over 1 year ago - 1 comment

#247 - Revert "Sweep Visualizations"

Pull Request - State: closed - Opened by norabelrose over 1 year ago - 1 comment

#246 - fix confusing comment

Pull Request - State: closed - Opened by lauritowal over 1 year ago

#245 - Sweep Visualizations

Pull Request - State: closed - Opened by derpyplops over 1 year ago - 2 comments

#244 - Ccs platt scale

Pull Request - State: closed - Opened by lauritowal over 1 year ago - 1 comment

#243 - add translated templates for imdb, boolq, truthfulqa

Pull Request - State: closed - Opened by ChristyKoh over 1 year ago - 2 comments

#242 - LEACE normalization

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#241 - Multiple choice datasets encounter a tensor error

Issue - State: open - Opened by ethanedwards over 1 year ago

#240 - log eval stats on train

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago

#239 - Don't left truncate stuff anymore

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#238 - Generalizable multi gpu to run e.g. Llama 65b

Pull Request - State: open - Opened by thejaminator over 1 year ago

#237 - add post_init checking for at least one dataset

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago

#236 - `burns` shortcut dataset in sweep

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#234 - Allow prompt templates to say if they should be binarized

Pull Request - State: closed - Opened by norabelrose over 1 year ago - 1 comment

#233 - Use spectral attribute removal for normalization

Pull Request - State: closed - Opened by norabelrose over 1 year ago - 1 comment

#232 - CCS results now as expected

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#231 - Load fp32 models in bfloat16 when possible

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#230 - [pre-commit.ci] pre-commit autoupdate

Pull Request - State: closed - Opened by pre-commit-ci[bot] over 1 year ago

#229 - Fix num_variants and num_shots arg

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago
Labels: bug

#228 - Hardcoded llama65 multigpu with accelerate

Pull Request - State: closed - Opened by thejaminator over 1 year ago

#227 - Support extraction in 8 bit with `bitsandbytes`

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#226 - Lots of simplification and refactoring

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#225 - wip: Test llama fsdp

Pull Request - State: closed - Opened by thejaminator over 1 year ago

#222 - Install is broken with python 3.11

Issue - State: closed - Opened by derpyplops over 1 year ago

#221 - WIP: Implement FSDP, drop usage of GeneratorBuilder, DIY caching

Pull Request - State: closed - Opened by thejaminator over 1 year ago

#219 - Reduce VINC reporter file size by >1000x

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago - 2 comments

#218 - Binarized meta-templates; some extraction refactoring

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#207 - [Draft] RWKV LM

Pull Request - State: open - Opened by Kyle1668 over 1 year ago - 1 comment

#203 - Use dill and Apache Arrow directly for caching & storing hidden states

Issue - State: open - Opened by norabelrose over 1 year ago - 1 comment
Labels: enhancement

#193 - Add way of specifying nondefault hparams that are shared across sweep

Issue - State: closed - Opened by lauritowal over 1 year ago - 1 comment

#192 - Add transfer eval to sweep

Issue - State: closed - Opened by lauritowal over 1 year ago - 1 comment

#187 - add custom_ds_n CLI arg

Pull Request - State: closed - Opened by reaganjlee over 1 year ago - 2 comments

#186 - Smoke tests for elk eval command

Issue - State: closed - Opened by norabelrose over 1 year ago - 4 comments

#180 - Add ELLMo Integration

Pull Request - State: closed - Opened by Kyle1668 over 1 year ago - 2 comments

#166 - check is streamable

Pull Request - State: closed - Opened by AlexTMallen over 1 year ago - 6 comments

#163 - Hyperparameter sweeps with Optuna or Ray Tune

Issue - State: closed - Opened by norabelrose over 1 year ago - 1 comment
Labels: enhancement

#157 - Gather eval stats

Pull Request - State: closed - Opened by reaganjlee over 1 year ago - 2 comments

#116 - Bootstrap CIs for AUROC metrics

Issue - State: closed - Opened by norabelrose over 1 year ago - 1 comment
Labels: enhancement

#113 - Can't save datasets with elk extract

Issue - State: closed - Opened by AlexTMallen over 1 year ago - 5 comments
Labels: bug

#100 - Move promptsource into a submodule; remove vestigial license

Pull Request - State: closed - Opened by norabelrose over 1 year ago

#99 - Fix single gpu stat logging

Pull Request - State: closed - Opened by thejaminator over 1 year ago - 1 comment

#98 - Add instructions for tests and typecheck

Pull Request - State: closed - Opened by thejaminator over 1 year ago

#97 - No `max_gpus` argument for `elk extract` command

Issue - State: closed - Opened by AlexTMallen over 1 year ago - 1 comment

#96 - Make normalization a property of the `Reporter`

Issue - State: closed - Opened by AlexTMallen over 1 year ago

#95 - Migrate to HF datasets for storing extracted hidden states

Pull Request - State: closed - Opened by norabelrose over 1 year ago - 2 comments
Labels: refactor

#94 - Testing reordering

Pull Request - State: closed - Opened by KaivuH almost 2 years ago

#93 - Add mypy, fix types for mypy

Pull Request - State: closed - Opened by thejaminator almost 2 years ago - 3 comments