Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / EleutherAI/lm-evaluation-harness issues and pull requests

#976 - Build failure?

Issue - State: closed - Opened by zhimin-z 11 months ago - 1 comment

#975 - Python 3.8 support on the main branch

Issue - State: closed - Opened by gugarosa 11 months ago - 2 comments

#973 - Same result on GPTQ 8bit and 4bit model, normal?

Issue - State: closed - Opened by Chrisz236 11 months ago - 8 comments
Labels: bug

#971 - [Refactor] add squad from master

Pull Request - State: closed - Opened by lintangsutawika 11 months ago

#970 - can you support trt-llm backends?

Issue - State: closed - Opened by jyjyjyjyjyjyj 11 months ago - 2 comments
Labels: feature request

#969 - [Refactor] Continuous Metrics

Pull Request - State: closed - Opened by lintangsutawika 11 months ago - 2 comments

#967 - [Refactor] Upstream ggml from big-refactor branch

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago - 4 comments

#965 - [Refactor] Remove deprecated `gold_alias` task YAML option

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago

#963 - [Refactor] Incorporate `version` field for tasks into `metadata`

Issue - State: closed - Opened by haileyschoelkopf 11 months ago
Labels: feature request

#962 - [Refactor] Allow for some tasks to force zero-shot

Issue - State: closed - Opened by haileyschoelkopf 11 months ago - 1 comment
Labels: feature request

#961 - gpt3.5-turbo

Issue - State: closed - Opened by ichitaka 11 months ago

#960 - globally normalized models

Issue - State: open - Opened by denizyuret 11 months ago - 1 comment

#959 - chatglm2 acc=0 on lambada_openai dataset, is it correct?

Issue - State: open - Opened by changwangss 11 months ago - 3 comments
Labels: bug

#958 - [Refactor] Verbosity rework

Pull Request - State: closed - Opened by lintangsutawika 11 months ago - 1 comment

#957 - [Refactor] Patch for Generation Until

Pull Request - State: closed - Opened by lintangsutawika 11 months ago

#956 - [Refactor] Describe local dataset usage in docs

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago

#955 - [Refactor] Update README, documentation

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago

#954 - [Refactor] Update documentation

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago

#953 - [Refactor] Don't load MMLU auxiliary_train set

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago

#952 - [Refactor] Logging fixes

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago - 1 comment

#951 - request pad not work as intended and potential solution

Issue - State: closed - Opened by ghost 11 months ago - 4 comments
Labels: bug

#950 - KeyError: 'mc1_targets' for truthfulqa_mc

Issue - State: closed - Opened by umarbeknasimov 11 months ago - 8 comments

#949 - [Refactor] Fix whitespace warning

Pull Request - State: closed - Opened by haileyschoelkopf 11 months ago - 1 comment

#948 - Dataset licenses - PR 2

Pull Request - State: closed - Opened by glerzing 11 months ago - 5 comments

#947 - [Refactor] Invalid DDP on multi-nodes single-card environment

Issue - State: closed - Opened by AndyWolfZwei 11 months ago - 1 comment

#946 - About the number of bbh task

Issue - State: closed - Opened by sglucas 11 months ago - 4 comments
Labels: bug

#945 - Loading mmlu eval not efficient

Issue - State: closed - Opened by zyh3826 11 months ago - 5 comments

#944 - Update scorer for TriviaQA task

Pull Request - State: open - Opened by vvchernov 11 months ago - 6 comments

#943 - Update scorer for gsm8k task

Pull Request - State: open - Opened by vvchernov 11 months ago

#941 - Prompt Structure

Issue - State: closed - Opened by YashSharma 11 months ago - 1 comment

#940 - 请问测评Qwen大模型的话需要使用什么参数

Issue - State: closed - Opened by dhh456 11 months ago

#939 - Loading GPTQ model, unexpected keyword argument 'quantized'

Issue - State: closed - Opened by noobmaster29 11 months ago - 1 comment

#938 - [refactor] squadv2 task results quite different than main branch

Issue - State: closed - Opened by emilyvanark 12 months ago - 8 comments

#937 - Big refactor write out adaption

Pull Request - State: closed - Opened by MicPie 12 months ago - 1 comment

#936 - [API] Add octoai back-end

Pull Request - State: open - Opened by vvchernov 12 months ago - 10 comments

#935 - [Refactor] Fix Default Metric Call

Pull Request - State: closed - Opened by lintangsutawika 12 months ago

#934 - [Refactor]fix two bugs when ran with qasper_bool and toxigen

Pull Request - State: closed - Opened by AndyWolfZwei 12 months ago - 4 comments

#933 - GPT2-XL does not work in ToxiGen

Issue - State: closed - Opened by Nkluge-correa 12 months ago - 3 comments
Labels: bug

#931 - [Refactor] Generate_until rename

Pull Request - State: closed - Opened by haileyschoelkopf 12 months ago - 1 comment

#929 - Fix `generate_until` rename

Pull Request - State: closed - Opened by haileyschoelkopf 12 months ago

#928 - PAWS-X Scores Significantly Differ between `master` vs `big-refactor`

Issue - State: closed - Opened by nitsanluke 12 months ago - 2 comments

#926 - [Refactor] Precommit formatting for Belebele

Pull Request - State: closed - Opened by lintangsutawika 12 months ago

#925 - Alternative Worlds Prompts for Various Tasks and Benchmarks

Pull Request - State: open - Opened by lintangsutawika 12 months ago - 3 comments

#924 - Hellaswag in other languages?

Issue - State: closed - Opened by Dex94 12 months ago - 3 comments

#923 - [Refactor] Squadv2 updates

Pull Request - State: closed - Opened by lintangsutawika 12 months ago

#922 - [Refactor] Mmlu subgroups and weight avg

Pull Request - State: closed - Opened by lintangsutawika 12 months ago - 14 comments

#921 - can we get validation loss?

Issue - State: closed - Opened by wanglamao2 12 months ago - 1 comment

#920 - Update Dockerfile

Pull Request - State: closed - Opened by luiscosio 12 months ago - 1 comment

#919 - Readme not clear

Issue - State: closed - Opened by aliasaria 12 months ago - 1 comment

#918 - pass through low_cpu_mem_usage

Pull Request - State: closed - Opened by Muennighoff 12 months ago

#917 - Update README.md

Pull Request - State: closed - Opened by StellaAthena 12 months ago

#915 - Update pyproject.toml

Pull Request - State: closed - Opened by lintangsutawika 12 months ago - 3 comments

#914 - Multiple-choice-fix

Pull Request - State: closed - Opened by keirp 12 months ago - 1 comment

#913 - Do not enforce unnecessary dependencies

Pull Request - State: closed - Opened by Muennighoff 12 months ago

#912 - [Refactor] Add _batch_scheduler in greedy_until

Pull Request - State: closed - Opened by AndyWolfZwei 12 months ago - 2 comments

#911 - GPT-2 Scores on lambada_openai Don't Match Paper

Issue - State: closed - Opened by jasonkrone 12 months ago - 6 comments
Labels: validation

#910 - [Refactor] Verbose

Pull Request - State: closed - Opened by lintangsutawika 12 months ago

#909 - Add support for GPU evaluation

Pull Request - State: closed - Opened by sgwhat 12 months ago - 1 comment

#908 - [Refactor] Improve error logging

Pull Request - State: closed - Opened by baberabb 12 months ago

#906 - [Refactor] Add MMLU task and fix a bug in greedy_until

Pull Request - State: closed - Opened by AndyWolfZwei 12 months ago - 8 comments

#905 - [Refactor] Fix Unit Tests

Pull Request - State: closed - Opened by haileyschoelkopf 12 months ago

#904 - Add `disable_exllama` model arg for HuggingFaceAutoLM

Pull Request - State: closed - Opened by codehound42 12 months ago - 1 comment

#903 - Evaluation on local dataset

Issue - State: closed - Opened by Anindyadeep 12 months ago - 2 comments

#898 - CoQA no longer works

Issue - State: closed - Opened by BlinkDL 12 months ago - 1 comment
Labels: bug

#897 - Allow Generation arguments on greedy_until reqs

Pull Request - State: closed - Opened by uSaiPrashanth 12 months ago - 3 comments

#896 - Prompt Templating

Issue - State: closed - Opened by sachith-surge 12 months ago - 6 comments

#894 - [big-refactor] Error in wildcards for task names

Issue - State: closed - Opened by nitsanluke 12 months ago

#892 - [big-refactor] Accelerate launch FSDP Runtime Error

Issue - State: closed - Opened by adamjackson2357 12 months ago - 7 comments

#891 - How to test with multiple Gpus

Issue - State: closed - Opened by mkkk1112 about 1 year ago - 4 comments

#887 - Werid evaluation result of MMLU

Issue - State: closed - Opened by Yuxin715d about 1 year ago - 4 comments
Labels: validation

#886 - PubMedQA evaluation is done incorrectly

Issue - State: open - Opened by tmabraham about 1 year ago

#885 - add belebele

Pull Request - State: closed - Opened by ManuelFay about 1 year ago - 15 comments

#884 - "RuntimeError: CUDA out of memory" on lm-eval 0.3.0 through GPT-NeoX evaluate past a certain number of nodes

Issue - State: open - Opened by AIproj about 1 year ago
Labels: bug, duplicate, help wanted

#883 - Add transformation filters

Pull Request - State: open - Opened by chrisociepa about 1 year ago

#882 - Add Belebele dataset

Pull Request - State: closed - Opened by ManuelFay about 1 year ago - 4 comments

#881 - Add mmlu average score in report

Issue - State: closed - Opened by Reason-Wang about 1 year ago - 5 comments
Labels: feature request, good first issue

#880 - [Refactor] Hotfixes to big-refactor

Pull Request - State: closed - Opened by haileyschoelkopf about 1 year ago - 1 comment

#879 - Potential Bug- Results change with batch size

Issue - State: closed - Opened by AbhinavDutta about 1 year ago - 3 comments

#878 - [Refactor] README.md for Asdiv

Pull Request - State: closed - Opened by lintangsutawika about 1 year ago

#877 - Fix positional arguments in HF model generate

Pull Request - State: closed - Opened by chrisociepa about 1 year ago - 1 comment

#876 - fix bug with output path in CWD

Pull Request - State: closed - Opened by jonabur about 1 year ago - 5 comments

#875 - [big refactor]mmlu is needed

Issue - State: closed - Opened by xiaol about 1 year ago - 1 comment

#874 - Calibration Features

Pull Request - State: closed - Opened by herbiebradley about 1 year ago - 3 comments

#873 - Stopping Criteria depending on the batch size leads to different results

Issue - State: closed - Opened by danielfleischer about 1 year ago - 5 comments
Labels: bug, help wanted

#872 - [Refactor] Scrolls

Pull Request - State: closed - Opened by lintangsutawika about 1 year ago - 1 comment

#871 - [Refactor] Deactivate select GH Actions

Pull Request - State: closed - Opened by haileyschoelkopf about 1 year ago

#870 - Create cot_yaml

Pull Request - State: closed - Opened by lintangsutawika about 1 year ago

#869 - TGI support - API evaluation of HF models

Issue - State: open - Opened by ManuelFay about 1 year ago - 10 comments
Labels: help wanted, feature request

#868 - Massive

Pull Request - State: closed - Opened by akjuneja about 1 year ago - 1 comment

#867 - How to make sure the harness utilizes ONLY the gpu mentioned with --devices

Issue - State: closed - Opened by AbhinavDutta about 1 year ago - 1 comment