Ecosyste.ms: Issues
An open API service for providing issue and pull request metadata for open source projects.
GitHub / EleutherAI/lm-evaluation-harness issues and pull requests
#976 - Build failure?
Issue -
State: closed - Opened by zhimin-z 11 months ago
- 1 comment
#975 - Python 3.8 support on the main branch
Issue -
State: closed - Opened by gugarosa 11 months ago
- 2 comments
#974 - toxigen task measures toxicity classification rather than whether generations are toxic?
Issue -
State: open - Opened by laphang 11 months ago
- 8 comments
#973 - Same result on GPTQ 8bit and 4bit model, normal?
Issue -
State: closed - Opened by Chrisz236 11 months ago
- 8 comments
Labels: bug
#972 - fine-tuning LLaMA on common sense reasoning datasets such as PIQA and HellaSwag
Issue -
State: closed - Opened by Ahmed-Roushdy 11 months ago
- 1 comment
#971 - [Refactor] add squad from master
Pull Request -
State: closed - Opened by lintangsutawika 11 months ago
#970 - can you support trt-llm backends?
Issue -
State: closed - Opened by jyjyjyjyjyjyj 11 months ago
- 2 comments
Labels: feature request
#969 - [Refactor] Continuous Metrics
Pull Request -
State: closed - Opened by lintangsutawika 11 months ago
- 2 comments
#968 - [API] Use private HF token for HF models and hide it when print to json file or console
Pull Request -
State: open - Opened by vvchernov 11 months ago
- 3 comments
#967 - [Refactor] Upstream ggml from big-refactor branch
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
- 4 comments
#966 - [Refactor] Bugfix: AttributeError: 'Namespace' object has no attribute 'verbose'
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
#965 - [Refactor] Remove deprecated `gold_alias` task YAML option
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
#964 - [Refactor] Remove `gold_alias` stale YAML config option
Issue -
State: closed - Opened by haileyschoelkopf 11 months ago
#963 - [Refactor] Incorporate `version` field for tasks into `metadata`
Issue -
State: closed - Opened by haileyschoelkopf 11 months ago
Labels: feature request
#962 - [Refactor] Allow for some tasks to force zero-shot
Issue -
State: closed - Opened by haileyschoelkopf 11 months ago
- 1 comment
Labels: feature request
#961 - gpt3.5-turbo
Issue -
State: closed - Opened by ichitaka 11 months ago
#960 - globally normalized models
Issue -
State: open - Opened by denizyuret 11 months ago
- 1 comment
#959 - chatglm2 acc=0 on lambada_openai dataset, is it correct?
Issue -
State: open - Opened by changwangss 11 months ago
- 3 comments
Labels: bug
#958 - [Refactor] Verbosity rework
Pull Request -
State: closed - Opened by lintangsutawika 11 months ago
- 1 comment
#957 - [Refactor] Patch for Generation Until
Pull Request -
State: closed - Opened by lintangsutawika 11 months ago
#956 - [Refactor] Describe local dataset usage in docs
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
#955 - [Refactor] Update README, documentation
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
#954 - [Refactor] Update documentation
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
#953 - [Refactor] Don't load MMLU auxiliary_train set
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
#952 - [Refactor] Logging fixes
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
- 1 comment
#951 - request pad not work as intended and potential solution
Issue -
State: closed - Opened by ghost 11 months ago
- 4 comments
Labels: bug
#950 - KeyError: 'mc1_targets' for truthfulqa_mc
Issue -
State: closed - Opened by umarbeknasimov 11 months ago
- 8 comments
#949 - [Refactor] Fix whitespace warning
Pull Request -
State: closed - Opened by haileyschoelkopf 11 months ago
- 1 comment
#948 - Dataset licenses - PR 2
Pull Request -
State: closed - Opened by glerzing 11 months ago
- 5 comments
#947 - [Refactor] Invalid DDP on multi-nodes single-card environment
Issue -
State: closed - Opened by AndyWolfZwei 11 months ago
- 1 comment
#946 - About the number of bbh task
Issue -
State: closed - Opened by sglucas 11 months ago
- 4 comments
Labels: bug
#945 - Loading mmlu eval not efficient
Issue -
State: closed - Opened by zyh3826 11 months ago
- 5 comments
#944 - Update scorer for TriviaQA task
Pull Request -
State: open - Opened by vvchernov 11 months ago
- 6 comments
#943 - Update scorer for gsm8k task
Pull Request -
State: open - Opened by vvchernov 11 months ago
#942 - Why is last token dropped in loglikelihood computation? Gives different result than when calculating loss.
Issue -
State: closed - Opened by sorenmulli 11 months ago
- 4 comments
#941 - Prompt Structure
Issue -
State: closed - Opened by YashSharma 11 months ago
- 1 comment
#940 - 请问测评Qwen大模型的话需要使用什么参数
Issue -
State: closed - Opened by dhh456 11 months ago
#939 - Loading GPTQ model, unexpected keyword argument 'quantized'
Issue -
State: closed - Opened by noobmaster29 11 months ago
- 1 comment
#938 - [refactor] squadv2 task results quite different than main branch
Issue -
State: closed - Opened by emilyvanark 12 months ago
- 8 comments
#937 - Big refactor write out adaption
Pull Request -
State: closed - Opened by MicPie 12 months ago
- 1 comment
#936 - [API] Add octoai back-end
Pull Request -
State: open - Opened by vvchernov 12 months ago
- 10 comments
#935 - [Refactor] Fix Default Metric Call
Pull Request -
State: closed - Opened by lintangsutawika 12 months ago
#934 - [Refactor]fix two bugs when ran with qasper_bool and toxigen
Pull Request -
State: closed - Opened by AndyWolfZwei 12 months ago
- 4 comments
#933 - GPT2-XL does not work in ToxiGen
Issue -
State: closed - Opened by Nkluge-correa 12 months ago
- 3 comments
Labels: bug
#932 - [Refactor] TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
Issue -
State: closed - Opened by emilyvanark 12 months ago
- 2 comments
#931 - [Refactor] Generate_until rename
Pull Request -
State: closed - Opened by haileyschoelkopf 12 months ago
- 1 comment
#930 - big-refactor branch, multi-GPU error:TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
Issue -
State: closed - Opened by mkkk1112 12 months ago
- 2 comments
#929 - Fix `generate_until` rename
Pull Request -
State: closed - Opened by haileyschoelkopf 12 months ago
#928 - PAWS-X Scores Significantly Differ between `master` vs `big-refactor`
Issue -
State: closed - Opened by nitsanluke 12 months ago
- 2 comments
#927 - [Refactor] change all mentions of `greedy_until` to `generate_until`
Pull Request -
State: closed - Opened by lintangsutawika 12 months ago
#926 - [Refactor] Precommit formatting for Belebele
Pull Request -
State: closed - Opened by lintangsutawika 12 months ago
#925 - Alternative Worlds Prompts for Various Tasks and Benchmarks
Pull Request -
State: open - Opened by lintangsutawika 12 months ago
- 3 comments
#924 - Hellaswag in other languages?
Issue -
State: closed - Opened by Dex94 12 months ago
- 3 comments
#923 - [Refactor] Squadv2 updates
Pull Request -
State: closed - Opened by lintangsutawika 12 months ago
#922 - [Refactor] Mmlu subgroups and weight avg
Pull Request -
State: closed - Opened by lintangsutawika 12 months ago
- 14 comments
#921 - can we get validation loss?
Issue -
State: closed - Opened by wanglamao2 12 months ago
- 1 comment
#920 - Update Dockerfile
Pull Request -
State: closed - Opened by luiscosio 12 months ago
- 1 comment
#919 - Readme not clear
Issue -
State: closed - Opened by aliasaria 12 months ago
- 1 comment
#918 - pass through low_cpu_mem_usage
Pull Request -
State: closed - Opened by Muennighoff 12 months ago
#917 - Update README.md
Pull Request -
State: closed - Opened by StellaAthena 12 months ago
#916 - Fix 'tqdm' object is not subscriptable" error in huggingface.py when batch size is auto
Pull Request -
State: closed - Opened by jasonkrone 12 months ago
- 1 comment
#915 - Update pyproject.toml
Pull Request -
State: closed - Opened by lintangsutawika 12 months ago
- 3 comments
#914 - Multiple-choice-fix
Pull Request -
State: closed - Opened by keirp 12 months ago
- 1 comment
#913 - Do not enforce unnecessary dependencies
Pull Request -
State: closed - Opened by Muennighoff 12 months ago
#912 - [Refactor] Add _batch_scheduler in greedy_until
Pull Request -
State: closed - Opened by AndyWolfZwei 12 months ago
- 2 comments
#911 - GPT-2 Scores on lambada_openai Don't Match Paper
Issue -
State: closed - Opened by jasonkrone 12 months ago
- 6 comments
Labels: validation
#910 - [Refactor] Verbose
Pull Request -
State: closed - Opened by lintangsutawika 12 months ago
#909 - Add support for GPU evaluation
Pull Request -
State: closed - Opened by sgwhat 12 months ago
- 1 comment
#908 - [Refactor] Improve error logging
Pull Request -
State: closed - Opened by baberabb 12 months ago
#906 - [Refactor] Add MMLU task and fix a bug in greedy_until
Pull Request -
State: closed - Opened by AndyWolfZwei 12 months ago
- 8 comments
#905 - [Refactor] Fix Unit Tests
Pull Request -
State: closed - Opened by haileyschoelkopf 12 months ago
#904 - Add `disable_exllama` model arg for HuggingFaceAutoLM
Pull Request -
State: closed - Opened by codehound42 12 months ago
- 1 comment
#903 - Evaluation on local dataset
Issue -
State: closed - Opened by Anindyadeep 12 months ago
- 2 comments
#898 - CoQA no longer works
Issue -
State: closed - Opened by BlinkDL 12 months ago
- 1 comment
Labels: bug
#897 - Allow Generation arguments on greedy_until reqs
Pull Request -
State: closed - Opened by uSaiPrashanth 12 months ago
- 3 comments
#896 - Prompt Templating
Issue -
State: closed - Opened by sachith-surge 12 months ago
- 6 comments
#894 - [big-refactor] Error in wildcards for task names
Issue -
State: closed - Opened by nitsanluke 12 months ago
#892 - [big-refactor] Accelerate launch FSDP Runtime Error
Issue -
State: closed - Opened by adamjackson2357 12 months ago
- 7 comments
#891 - How to test with multiple Gpus
Issue -
State: closed - Opened by mkkk1112 about 1 year ago
- 4 comments
#887 - Werid evaluation result of MMLU
Issue -
State: closed - Opened by Yuxin715d about 1 year ago
- 4 comments
Labels: validation
#886 - PubMedQA evaluation is done incorrectly
Issue -
State: open - Opened by tmabraham about 1 year ago
#885 - add belebele
Pull Request -
State: closed - Opened by ManuelFay about 1 year ago
- 15 comments
#884 - "RuntimeError: CUDA out of memory" on lm-eval 0.3.0 through GPT-NeoX evaluate past a certain number of nodes
Issue -
State: open - Opened by AIproj about 1 year ago
Labels: bug, duplicate, help wanted
#883 - Add transformation filters
Pull Request -
State: open - Opened by chrisociepa about 1 year ago
#882 - Add Belebele dataset
Pull Request -
State: closed - Opened by ManuelFay about 1 year ago
- 4 comments
#881 - Add mmlu average score in report
Issue -
State: closed - Opened by Reason-Wang about 1 year ago
- 5 comments
Labels: feature request, good first issue
#880 - [Refactor] Hotfixes to big-refactor
Pull Request -
State: closed - Opened by haileyschoelkopf about 1 year ago
- 1 comment
#879 - Potential Bug- Results change with batch size
Issue -
State: closed - Opened by AbhinavDutta about 1 year ago
- 3 comments
#878 - [Refactor] README.md for Asdiv
Pull Request -
State: closed - Opened by lintangsutawika about 1 year ago
#877 - Fix positional arguments in HF model generate
Pull Request -
State: closed - Opened by chrisociepa about 1 year ago
- 1 comment
#876 - fix bug with output path in CWD
Pull Request -
State: closed - Opened by jonabur about 1 year ago
- 5 comments
#875 - [big refactor]mmlu is needed
Issue -
State: closed - Opened by xiaol about 1 year ago
- 1 comment
#874 - Calibration Features
Pull Request -
State: closed - Opened by herbiebradley about 1 year ago
- 3 comments
#873 - Stopping Criteria depending on the batch size leads to different results
Issue -
State: closed - Opened by danielfleischer about 1 year ago
- 5 comments
Labels: bug, help wanted
#872 - [Refactor] Scrolls
Pull Request -
State: closed - Opened by lintangsutawika about 1 year ago
- 1 comment
#871 - [Refactor] Deactivate select GH Actions
Pull Request -
State: closed - Opened by haileyschoelkopf about 1 year ago
#870 - Create cot_yaml
Pull Request -
State: closed - Opened by lintangsutawika about 1 year ago
#869 - TGI support - API evaluation of HF models
Issue -
State: open - Opened by ManuelFay about 1 year ago
- 10 comments
Labels: help wanted, feature request
#868 - Massive
Pull Request -
State: closed - Opened by akjuneja about 1 year ago
- 1 comment
#867 - How to make sure the harness utilizes ONLY the gpu mentioned with --devices
Issue -
State: closed - Opened by AbhinavDutta about 1 year ago
- 1 comment