Ecosyste.ms: Issues
An open API service for providing issue and pull request metadata for open source projects.
GitHub / tatsu-lab/alpaca_eval issues and pull requests
#412 - add Self-taught-llama3.1-70B-dpo as a evaluator
Pull Request -
State: closed - Opened by tianlu-wang 6 days ago
- 1 comment
#411 - Add SelfMoA_gemma-2-9b-it-SimPO, SelfMoA_gemma-2-9b-it-WPO-HB to AlpacaEval
Pull Request -
State: closed - Opened by wenzhe-li 7 days ago
#410 - Will the annotator alpaca_eval_gpt4_turbo_fn also change?
Issue -
State: closed - Opened by hsqmlzno1 8 days ago
- 1 comment
#409 - Updated HF Link in model_configs for Llama-3-8B-Instruct-SkillMix
Pull Request -
State: closed - Opened by parksimon0808 14 days ago
#408 - Question about Leaderboard
Issue -
State: closed - Opened by MaoXinn 17 days ago
- 1 comment
#407 - I can't duplicate your results locally
Issue -
State: closed - Opened by suanflower 19 days ago
- 2 comments
#406 - 'message': "Invalid function definition for 'make_partial_leaderboard': unexpected parameter 'strict' parameter supplied
Issue -
State: closed - Opened by suanflower 20 days ago
- 1 comment
#405 - Add Llama-3-8B-Instruct-SkillMix to AlpacaEval
Pull Request -
State: closed - Opened by parksimon0808 22 days ago
- 1 comment
#404 - add example for Llama3 vllm server
Pull Request -
State: open - Opened by cameron-chen 22 days ago
- 1 comment
#403 - Add REBEL-Llama-3-8B-Instruct-Armo to AlpacaEval
Pull Request -
State: closed - Opened by ZhaolinGao about 1 month ago
- 1 comment
#402 - [ENH] add metadata to completion: date, version,...
Pull Request -
State: closed - Opened by YannDubs about 1 month ago
#401 - Add evaluator weighted_alpaca_eval_gpt-4o-mini-2024-07-18
Pull Request -
State: closed - Opened by tongyx361 about 1 month ago
- 1 comment
#400 - How to compute winrate using a file of 'annotations.json'
Issue -
State: closed - Opened by JinYujie99 about 1 month ago
#399 - Add blendaxai-gm-l6-vo31 to AlpacaEval
Pull Request -
State: closed - Opened by ym-blendax-ai about 1 month ago
- 1 comment
#398 - Add Shopee-SlimMoA-v1 to AlpacaEval
Pull Request -
State: closed - Opened by LLM-Alignment-sh about 1 month ago
- 1 comment
#397 - Add blendaxai-gm-l6-vo14 to AlpacaEval
Pull Request -
State: closed - Opened by ym-blendax-ai about 1 month ago
#396 - After I finished evaluating Meta-Llama3-8B-Instruct
Issue -
State: closed - Opened by iGangao about 1 month ago
- 2 comments
#395 - Added Llama3-PBM-Nova-70B model
Pull Request -
State: closed - Opened by PKU-Baichuan about 1 month ago
#394 - [ENH] add strict decoding OAI
Pull Request -
State: closed - Opened by YannDubs about 1 month ago
#393 - [ENH] add mistral v0.3, Qwen2 70b, gtp4 mini
Pull Request -
State: closed - Opened by YannDubs about 1 month ago
#392 - [ENH] enable base_dir to be a list
Pull Request -
State: closed - Opened by YannDubs about 2 months ago
#391 - [ENH] OpenAI use tools instead of functions
Pull Request -
State: closed - Opened by YannDubs about 2 months ago
#390 - all the following results are about AlpacaEval 1.0 and have not been updated since?
Issue -
State: closed - Opened by zhimin-z about 2 months ago
- 1 comment
#389 - Add blendaxai-gm-l3-v35 to AlpacaEval
Pull Request -
State: closed - Opened by ym-blendax-ai about 2 months ago
- 2 comments
#388 - Added Blendax.AI-gm-l3-v1 model results
Pull Request -
State: closed - Opened by ym-blendax-ai about 2 months ago
- 1 comment
#387 - Change the name of the Infinity-Instruct-7M-0729-Models to Infinity-Instruct-7M-Gen-Models
Pull Request -
State: closed - Opened by cszhengyh about 2 months ago
#386 - Change the name of the Infinity-Instruct-7M-0729-Models to Infinity-Instruct-7M-Gen-Models
Pull Request -
State: closed - Opened by cszhengyh about 2 months ago
#385 - Add link to gemma-2-9b-it-WPO-HB
Pull Request -
State: closed - Opened by wzhouad about 2 months ago
- 3 comments
#384 - Add gemma-2-9b-it-WPO-HB to AlpacaEval
Pull Request -
State: closed - Opened by wzhouad about 2 months ago
#383 - Add Infinity-Instruct-7M-0729-Llama3_1-70B, Infinity-Instruct-7M-0729-Llama3_1-8B, Infinity-Instruct-7M-0729-mistral-7B to AlpacaEval
Pull Request -
State: closed - Opened by cszhengyh about 2 months ago
- 1 comment
#382 - Non-reproducible results
Issue -
State: closed - Opened by felipemaiapolo 2 months ago
- 4 comments
#381 - [ENH] add example for LLama 3 vllm
Pull Request -
State: closed - Opened by YannDubs 2 months ago
#380 - Using local models as evaluators
Issue -
State: closed - Opened by luciolcv 2 months ago
- 2 comments
#379 - error with default json - ValueError: cannot reindex on an axis with duplicate labels
Issue -
State: closed - Opened by ishapuri 2 months ago
- 2 comments
#378 - [ENH] add llama 3.1
Pull Request -
State: closed - Opened by YannDubs 2 months ago
#377 - Add Llama-3-Instruct-8B-WPO-HB-v2 to AlpacaEval
Pull Request -
State: closed - Opened by wzhouad 2 months ago
- 1 comment
#376 - Questions about the 'use_beam_search' configuration.
Issue -
State: closed - Opened by ChangyuChen347 2 months ago
- 1 comment
#375 - Extra '\n' in the prompt of Llama-3-Instruct-8B-SimPO
Issue -
State: closed - Opened by gohsyi 2 months ago
- 3 comments
#374 - Why do we set do_sample to true during evaluation?
Issue -
State: closed - Opened by gohsyi 2 months ago
- 1 comment
#373 - [BUG] backward compatibility vllm do_sample -> use_beam_search
Pull Request -
State: closed - Opened by YannDubs 2 months ago
#372 - [ENH] adding simplified glm
Pull Request -
State: closed - Opened by YannDubs 2 months ago
#371 - [ENH] add the code to compute instruction_following
Pull Request -
State: closed - Opened by YannDubs 2 months ago
#370 - update model links
Pull Request -
State: closed - Opened by xiamengzhou 2 months ago
#369 - [ENH] add CI test for unwanted files
Pull Request -
State: closed - Opened by YannDubs 2 months ago
#368 - Add gemma-2-9b-it-SimPO and gemma-2-9b-it-DPO to AlpacaEval
Pull Request -
State: closed - Opened by xiamengzhou 3 months ago
- 4 comments
#367 - Add Higgs Llama3-70B V2 Results
Pull Request -
State: closed - Opened by sxjscience 3 months ago
- 2 comments
#366 - Added Ghost 8B Beta (d0x5) model
Pull Request -
State: closed - Opened by lh0x00 3 months ago
- 4 comments
#365 - assert output_path is not None and name in leaderboard
Issue -
State: closed - Opened by AIR-hl 3 months ago
- 1 comment
#364 - Add Infinity-Instruct-3M-0625-Models to AlpacaEval
Pull Request -
State: closed - Opened by cszhengyh 3 months ago
- 1 comment
#363 - Argument `do_sample` is removed from `vllm_local_completions`
Issue -
State: closed - Opened by xiamengzhou 3 months ago
- 1 comment
#361 - Negative Correlation while using `alpaca_eval analyze_evaluators`
Issue -
State: closed - Opened by EganGu 3 months ago
- 1 comment
#360 - Encountering Error about cannot reindex on an axis with duplicate labels
Issue -
State: closed - Opened by URRealHero 3 months ago
- 2 comments
#359 - Add SPPO-Gemma-2-9B-It-PairRM to AlpacaEval
Pull Request -
State: closed - Opened by angelahzyuan 3 months ago
#358 - Add Infinity-Instruct-3M-0613-Llama3-70B to AlpacaEval
Pull Request -
State: closed - Opened by cszhengyh 3 months ago
#357 - Add Infinity-Instruct-3M-0613-Llama3-70B to AlpacaEval
Pull Request -
State: closed - Opened by cszhengyh 3 months ago
#356 - Add Infinity-Instruct-3M-0613-Llama3-70B to AlpacaEval
Pull Request -
State: closed - Opened by cszhengyh 3 months ago
- 3 comments
#355 - AssertionError
Issue -
State: closed - Opened by BBaekdabang 3 months ago
#354 - Add SPPO-Llama-3-Instruct-8B-PairRM to AlpacaEval
Pull Request -
State: closed - Opened by Edward-Sun 3 months ago
- 1 comment
#353 - ValueError: Trailing data
Issue -
State: closed - Opened by BBaekdabang 3 months ago
- 3 comments
#352 - Discrepancy between alpaca leaderboard and Chatbot arena ELO
Issue -
State: closed - Opened by Varun221 3 months ago
- 1 comment
#351 - Add Infinity-Instruct-3M-0613-Mistral-7B to AlpacaEval
Pull Request -
State: closed - Opened by cszhengyh 3 months ago
- 1 comment
#350 - cannot reindex
Issue -
State: closed - Opened by Chirobocea 3 months ago
- 3 comments
#349 - [BUG] trust repo alpaca_eval
Pull Request -
State: closed - Opened by YannDubs 3 months ago
#348 - Add claude-3-5-sonnet-20240620 to AlpacaEval
Pull Request -
State: closed - Opened by MarjovanLier 3 months ago
- 1 comment
#347 - Add OpenPipe Mixture of Agents model to Alpaca Eval
Pull Request -
State: closed - Opened by saum7800 3 months ago
#346 - Details on Training GLM for Length-Controlled Winrate
Issue -
State: closed - Opened by yix8 3 months ago
- 3 comments
#345 - Add Nanbeige2-16B-Chat to AlpacaEval
Pull Request -
State: closed - Opened by yuani114 3 months ago
- 1 comment
#344 - Add Storm-7B, Storm-7B (best-of-64) to AlpacaEval
Pull Request -
State: closed - Opened by yifan123 4 months ago
- 3 comments
#343 - Evaluating a self hosted LLM through an API.
Issue -
State: closed - Opened by CHRISTINEMUTHEE 4 months ago
- 1 comment
#342 - Add Together-MoA, Together-MoA-Lite to AlpacaEval
Pull Request -
State: closed - Opened by IsThatYou 4 months ago
- 1 comment
#341 - ERROR:root:Error while parsing completion:
Issue -
State: closed - Opened by AGTSAAA 4 months ago
- 1 comment
#340 - How to solve the problem of null appearing in the evaluation results? Thank you very much
Issue -
State: closed - Opened by AGTSAAA 4 months ago
- 3 comments
#339 - tensor_parallel_size can not work
Issue -
State: closed - Opened by AGTSAAA 4 months ago
- 1 comment
#338 - [BUG] fix bs in VLLM and add chatml
Pull Request -
State: closed - Opened by YannDubs 4 months ago
#337 - Why is max_num_seqs allowed here?
Issue -
State: closed - Opened by RAY2L 4 months ago
- 1 comment
#336 - Preference doesn't match log_probs in `annotations.json`
Issue -
State: closed - Opened by YJWon99 4 months ago
- 1 comment
#335 - confused about openai API
Issue -
State: closed - Opened by junkangwu 4 months ago
- 1 comment
#334 - Add merlinite-7B-AOT to AlpacaEval
Pull Request -
State: closed - Opened by imelnyk 4 months ago
- 1 comment
#333 - The code for computing instruction difficulty
Issue -
State: closed - Opened by calvinh99 4 months ago
- 1 comment
#332 - fix model link
Pull Request -
State: closed - Opened by chujiezheng 4 months ago
#331 - Add ExPO + `Llama-3-Instruct-8B-SimPO` results
Pull Request -
State: closed - Opened by chujiezheng 4 months ago
- 1 comment
#330 - [ENH&BUG] improve VLLM
Pull Request -
State: closed - Opened by YannDubs 4 months ago
#329 - Trouble with custom model hosted on OpenAI compatible endpoint
Issue -
State: closed - Opened by tastycode 4 months ago
- 1 comment
#328 - Unexpected low judge preference for some prompts
Issue -
State: closed - Opened by geoalgo 4 months ago
- 1 comment
#327 - Question on assumption of `model_identity` as a factor for preference on generated outputs.
Issue -
State: closed - Opened by fgenie 4 months ago
- 3 comments
#326 - Add REBEL-Llama-3-8B-Instruct to AlpacaEval
Pull Request -
State: closed - Opened by ZhaolinGao 4 months ago
- 1 comment
#325 - Huge performance gap when using annotator weighted_alpaca_eval_gpt4_turbo and alpaca_eval_gpt4_turbo_fn
Issue -
State: closed - Opened by ZeroYuHuang 4 months ago
- 4 comments
#324 - Add Aligner 2B+GPT-4 Turbo (04/09) Results
Pull Request -
State: closed - Opened by AlignInc 4 months ago
- 1 comment
#323 - Add Aligner 2B+GPT-4 Turbo (04/09) to AlpacaEval
Pull Request -
State: closed - Opened by AlignInc 4 months ago
#322 - Add Phi 3 models
Issue -
State: closed - Opened by EwoutH 4 months ago
- 3 comments
#321 - [ENH] Use multi threading instead of processing
Pull Request -
State: closed - Opened by YannDubs 4 months ago
#320 - Add Llama-3-Instruct-8B-SimPO to AlpacaEval
Pull Request -
State: closed - Opened by xiamengzhou 5 months ago
- 1 comment
#319 - [ENH] vicuna 1.5
Pull Request -
State: closed - Opened by YannDubs 5 months ago
#318 - [CLEAN] move evaluators lb llama3
Pull Request -
State: closed - Opened by YannDubs 5 months ago
#317 - [ENH] add LC SEM
Pull Request -
State: closed - Opened by YannDubs 5 months ago
#316 - Alpaca Evaluation Instruction Difficulty used also for Custom Evaluation Dataset
Issue -
State: closed - Opened by fanconic 5 months ago
- 3 comments
#315 - Update README.md
Pull Request -
State: closed - Opened by zhuang-li 5 months ago
- 1 comment
#314 - llama3 evaluator
Pull Request -
State: closed - Opened by zhuang-li 5 months ago
- 2 comments
#313 - possibility of adding llama3-70b as the evaluator?
Issue -
State: closed - Opened by zhuang-li 5 months ago
- 3 comments