Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / tatsu-lab/alpaca_eval issues and pull requests

#412 - add Self-taught-llama3.1-70B-dpo as a evaluator

Pull Request - State: closed - Opened by tianlu-wang 6 days ago - 1 comment

#410 - Will the annotator alpaca_eval_gpt4_turbo_fn also change?

Issue - State: closed - Opened by hsqmlzno1 8 days ago - 1 comment

#408 - Question about Leaderboard

Issue - State: closed - Opened by MaoXinn 17 days ago - 1 comment

#407 - I can't duplicate your results locally

Issue - State: closed - Opened by suanflower 19 days ago - 2 comments

#405 - Add Llama-3-8B-Instruct-SkillMix to AlpacaEval

Pull Request - State: closed - Opened by parksimon0808 22 days ago - 1 comment

#404 - add example for Llama3 vllm server

Pull Request - State: open - Opened by cameron-chen 22 days ago - 1 comment

#403 - Add REBEL-Llama-3-8B-Instruct-Armo to AlpacaEval

Pull Request - State: closed - Opened by ZhaolinGao about 1 month ago - 1 comment

#402 - [ENH] add metadata to completion: date, version,...

Pull Request - State: closed - Opened by YannDubs about 1 month ago

#401 - Add evaluator weighted_alpaca_eval_gpt-4o-mini-2024-07-18

Pull Request - State: closed - Opened by tongyx361 about 1 month ago - 1 comment

#400 - How to compute winrate using a file of 'annotations.json'

Issue - State: closed - Opened by JinYujie99 about 1 month ago

#399 - Add blendaxai-gm-l6-vo31 to AlpacaEval

Pull Request - State: closed - Opened by ym-blendax-ai about 1 month ago - 1 comment

#398 - Add Shopee-SlimMoA-v1 to AlpacaEval

Pull Request - State: closed - Opened by LLM-Alignment-sh about 1 month ago - 1 comment

#397 - Add blendaxai-gm-l6-vo14 to AlpacaEval

Pull Request - State: closed - Opened by ym-blendax-ai about 1 month ago

#396 - After I finished evaluating Meta-Llama3-8B-Instruct

Issue - State: closed - Opened by iGangao about 1 month ago - 2 comments

#395 - Added Llama3-PBM-Nova-70B model

Pull Request - State: closed - Opened by PKU-Baichuan about 1 month ago

#394 - [ENH] add strict decoding OAI

Pull Request - State: closed - Opened by YannDubs about 1 month ago

#393 - [ENH] add mistral v0.3, Qwen2 70b, gtp4 mini

Pull Request - State: closed - Opened by YannDubs about 1 month ago

#392 - [ENH] enable base_dir to be a list

Pull Request - State: closed - Opened by YannDubs about 2 months ago

#391 - [ENH] OpenAI use tools instead of functions

Pull Request - State: closed - Opened by YannDubs about 2 months ago

#389 - Add blendaxai-gm-l3-v35 to AlpacaEval

Pull Request - State: closed - Opened by ym-blendax-ai about 2 months ago - 2 comments

#388 - Added Blendax.AI-gm-l3-v1 model results

Pull Request - State: closed - Opened by ym-blendax-ai about 2 months ago - 1 comment

#385 - Add link to gemma-2-9b-it-WPO-HB

Pull Request - State: closed - Opened by wzhouad about 2 months ago - 3 comments

#384 - Add gemma-2-9b-it-WPO-HB to AlpacaEval

Pull Request - State: closed - Opened by wzhouad about 2 months ago

#382 - Non-reproducible results

Issue - State: closed - Opened by felipemaiapolo 2 months ago - 4 comments

#381 - [ENH] add example for LLama 3 vllm

Pull Request - State: closed - Opened by YannDubs 2 months ago

#380 - Using local models as evaluators

Issue - State: closed - Opened by luciolcv 2 months ago - 2 comments

#378 - [ENH] add llama 3.1

Pull Request - State: closed - Opened by YannDubs 2 months ago

#377 - Add Llama-3-Instruct-8B-WPO-HB-v2 to AlpacaEval

Pull Request - State: closed - Opened by wzhouad 2 months ago - 1 comment

#376 - Questions about the 'use_beam_search' configuration.

Issue - State: closed - Opened by ChangyuChen347 2 months ago - 1 comment

#375 - Extra '\n' in the prompt of Llama-3-Instruct-8B-SimPO

Issue - State: closed - Opened by gohsyi 2 months ago - 3 comments

#374 - Why do we set do_sample to true during evaluation?

Issue - State: closed - Opened by gohsyi 2 months ago - 1 comment

#373 - [BUG] backward compatibility vllm do_sample -> use_beam_search

Pull Request - State: closed - Opened by YannDubs 2 months ago

#372 - [ENH] adding simplified glm

Pull Request - State: closed - Opened by YannDubs 2 months ago

#371 - [ENH] add the code to compute instruction_following

Pull Request - State: closed - Opened by YannDubs 2 months ago

#370 - update model links

Pull Request - State: closed - Opened by xiamengzhou 2 months ago

#369 - [ENH] add CI test for unwanted files

Pull Request - State: closed - Opened by YannDubs 2 months ago

#368 - Add gemma-2-9b-it-SimPO and gemma-2-9b-it-DPO to AlpacaEval

Pull Request - State: closed - Opened by xiamengzhou 3 months ago - 4 comments

#367 - Add Higgs Llama3-70B V2 Results

Pull Request - State: closed - Opened by sxjscience 3 months ago - 2 comments

#366 - Added Ghost 8B Beta (d0x5) model

Pull Request - State: closed - Opened by lh0x00 3 months ago - 4 comments

#365 - assert output_path is not None and name in leaderboard

Issue - State: closed - Opened by AIR-hl 3 months ago - 1 comment

#364 - Add Infinity-Instruct-3M-0625-Models to AlpacaEval

Pull Request - State: closed - Opened by cszhengyh 3 months ago - 1 comment

#363 - Argument `do_sample` is removed from `vllm_local_completions`

Issue - State: closed - Opened by xiamengzhou 3 months ago - 1 comment

#362 - .

Issue - State: closed - Opened by BeastyZ 3 months ago - 2 comments

#361 - Negative Correlation while using `alpaca_eval analyze_evaluators`

Issue - State: closed - Opened by EganGu 3 months ago - 1 comment

#360 - Encountering Error about cannot reindex on an axis with duplicate labels

Issue - State: closed - Opened by URRealHero 3 months ago - 2 comments

#359 - Add SPPO-Gemma-2-9B-It-PairRM to AlpacaEval

Pull Request - State: closed - Opened by angelahzyuan 3 months ago

#358 - Add Infinity-Instruct-3M-0613-Llama3-70B to AlpacaEval

Pull Request - State: closed - Opened by cszhengyh 3 months ago

#357 - Add Infinity-Instruct-3M-0613-Llama3-70B to AlpacaEval

Pull Request - State: closed - Opened by cszhengyh 3 months ago

#356 - Add Infinity-Instruct-3M-0613-Llama3-70B to AlpacaEval

Pull Request - State: closed - Opened by cszhengyh 3 months ago - 3 comments

#355 - AssertionError

Issue - State: closed - Opened by BBaekdabang 3 months ago

#354 - Add SPPO-Llama-3-Instruct-8B-PairRM to AlpacaEval

Pull Request - State: closed - Opened by Edward-Sun 3 months ago - 1 comment

#353 - ValueError: Trailing data

Issue - State: closed - Opened by BBaekdabang 3 months ago - 3 comments

#352 - Discrepancy between alpaca leaderboard and Chatbot arena ELO

Issue - State: closed - Opened by Varun221 3 months ago - 1 comment

#351 - Add Infinity-Instruct-3M-0613-Mistral-7B to AlpacaEval

Pull Request - State: closed - Opened by cszhengyh 3 months ago - 1 comment

#350 - cannot reindex

Issue - State: closed - Opened by Chirobocea 3 months ago - 3 comments

#349 - [BUG] trust repo alpaca_eval

Pull Request - State: closed - Opened by YannDubs 3 months ago

#348 - Add claude-3-5-sonnet-20240620 to AlpacaEval

Pull Request - State: closed - Opened by MarjovanLier 3 months ago - 1 comment

#347 - Add OpenPipe Mixture of Agents model to Alpaca Eval

Pull Request - State: closed - Opened by saum7800 3 months ago

#346 - Details on Training GLM for Length-Controlled Winrate

Issue - State: closed - Opened by yix8 3 months ago - 3 comments

#345 - Add Nanbeige2-16B-Chat to AlpacaEval

Pull Request - State: closed - Opened by yuani114 3 months ago - 1 comment

#344 - Add Storm-7B, Storm-7B (best-of-64) to AlpacaEval

Pull Request - State: closed - Opened by yifan123 4 months ago - 3 comments

#343 - Evaluating a self hosted LLM through an API.

Issue - State: closed - Opened by CHRISTINEMUTHEE 4 months ago - 1 comment

#342 - Add Together-MoA, Together-MoA-Lite to AlpacaEval

Pull Request - State: closed - Opened by IsThatYou 4 months ago - 1 comment

#341 - ERROR:root:Error while parsing completion:

Issue - State: closed - Opened by AGTSAAA 4 months ago - 1 comment

#339 - tensor_parallel_size can not work

Issue - State: closed - Opened by AGTSAAA 4 months ago - 1 comment

#338 - [BUG] fix bs in VLLM and add chatml

Pull Request - State: closed - Opened by YannDubs 4 months ago

#337 - Why is max_num_seqs allowed here?

Issue - State: closed - Opened by RAY2L 4 months ago - 1 comment

#336 - Preference doesn't match log_probs in `annotations.json`

Issue - State: closed - Opened by YJWon99 4 months ago - 1 comment

#335 - confused about openai API

Issue - State: closed - Opened by junkangwu 4 months ago - 1 comment

#334 - Add merlinite-7B-AOT to AlpacaEval

Pull Request - State: closed - Opened by imelnyk 4 months ago - 1 comment

#333 - The code for computing instruction difficulty

Issue - State: closed - Opened by calvinh99 4 months ago - 1 comment

#332 - fix model link

Pull Request - State: closed - Opened by chujiezheng 4 months ago

#331 - Add ExPO + `Llama-3-Instruct-8B-SimPO` results

Pull Request - State: closed - Opened by chujiezheng 4 months ago - 1 comment

#330 - [ENH&BUG] improve VLLM

Pull Request - State: closed - Opened by YannDubs 4 months ago

#329 - Trouble with custom model hosted on OpenAI compatible endpoint

Issue - State: closed - Opened by tastycode 4 months ago - 1 comment

#328 - Unexpected low judge preference for some prompts

Issue - State: closed - Opened by geoalgo 4 months ago - 1 comment

#326 - Add REBEL-Llama-3-8B-Instruct to AlpacaEval

Pull Request - State: closed - Opened by ZhaolinGao 4 months ago - 1 comment

#324 - Add Aligner 2B+GPT-4 Turbo (04/09) Results

Pull Request - State: closed - Opened by AlignInc 4 months ago - 1 comment

#323 - Add Aligner 2B+GPT-4 Turbo (04/09) to AlpacaEval

Pull Request - State: closed - Opened by AlignInc 4 months ago

#322 - Add Phi 3 models

Issue - State: closed - Opened by EwoutH 4 months ago - 3 comments

#321 - [ENH] Use multi threading instead of processing

Pull Request - State: closed - Opened by YannDubs 4 months ago

#320 - Add Llama-3-Instruct-8B-SimPO to AlpacaEval

Pull Request - State: closed - Opened by xiamengzhou 5 months ago - 1 comment

#319 - [ENH] vicuna 1.5

Pull Request - State: closed - Opened by YannDubs 5 months ago

#318 - [CLEAN] move evaluators lb llama3

Pull Request - State: closed - Opened by YannDubs 5 months ago

#317 - [ENH] add LC SEM

Pull Request - State: closed - Opened by YannDubs 5 months ago

#315 - Update README.md

Pull Request - State: closed - Opened by zhuang-li 5 months ago - 1 comment

#314 - llama3 evaluator

Pull Request - State: closed - Opened by zhuang-li 5 months ago - 2 comments

#313 - possibility of adding llama3-70b as the evaluator?

Issue - State: closed - Opened by zhuang-li 5 months ago - 3 comments