Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / embeddings-benchmark/mteb issues and pull requests

#1845 - Add gte-modernbert-base

Pull Request - State: open - Opened by afalf 17 days ago

#1844 - Support for MTEB-ES (Spanish Benchmark)

Issue - State: open - Opened by elsatch 17 days ago - 2 comments
Labels: enhancement, new benchmark

#1843 - Add FaMTEB (Farsi/Persian Text Embedding Benchmark)

Pull Request - State: closed - Opened by mehran-sarmadi 17 days ago - 12 comments

#1842 - add zero-shot definition to new LB

Issue - State: closed - Opened by Muennighoff 17 days ago - 3 comments
Labels: leaderboard

#1841 - New leaderboard filtering cross-encoders

Issue - State: open - Opened by Muennighoff 17 days ago - 6 comments
Labels: leaderboard

#1840 - Missing results on new leaderboard

Issue - State: closed - Opened by Muennighoff 17 days ago - 3 comments
Labels: leaderboard

#1839 - nan in leaderboard

Issue - State: closed - Opened by Muennighoff 17 days ago - 2 comments
Labels: leaderboard

#1838 - Leaderboard: Make benchmark not `beta` and give them better descriptions

Issue - State: closed - Opened by x-tabdeveloping 18 days ago - 1 comment
Labels: leaderboard

#1836 - Add LOTTE

Issue - State: open - Opened by Samoed 18 days ago
Labels: new benchmark

#1833 - Add more annotations

Pull Request - State: closed - Opened by Samoed 19 days ago - 1 comment

#1832 - [v2] Remove MultilingualTask

Issue - State: closed - Opened by Samoed 20 days ago - 1 comment
Labels: v2

#1831 - Optimize the slowest test for better performance

Issue - State: open - Opened by sam-hey 20 days ago - 3 comments

#1830 - fix: subsets to run

Pull Request - State: closed - Opened by Samoed 20 days ago - 1 comment

#1829 - [v2] ci: run bm25 and ColBERT test in ci

Pull Request - State: closed - Opened by sam-hey 20 days ago - 10 comments

#1828 - Remove embedding layers when calculating number of parameters

Issue - State: open - Opened by KennethEnevoldsen 20 days ago
Labels: enhancement, leaderboard

#1827 - fix: bm25s

Pull Request - State: closed - Opened by sam-hey 20 days ago

#1826 - Generalize Custom search() Method

Issue - State: open - Opened by sam-hey 20 days ago - 8 comments

#1825 - [mieb] Investigate voc2007 vlm2vec

Pull Request - State: closed - Opened by isaac-chung 21 days ago - 1 comment

#1824 - fix: Added Chinese Stella models

Pull Request - State: closed - Opened by x-tabdeveloping 22 days ago - 3 comments

#1823 - [MIEB] migrate results from tmp repo to results repo

Issue - State: open - Opened by isaac-chung 22 days ago - 1 comment
Labels: mieb

#1822 - [v2] Remove `LANG_MAPPING`

Issue - State: open - Opened by Samoed 22 days ago
Labels: v2

#1821 - fix: add bge-m3 `ModelMeta`

Pull Request - State: closed - Opened by Samoed 22 days ago - 1 comment

#1820 - [v2] remove metadata_dict

Pull Request - State: closed - Opened by Samoed 22 days ago

#1819 - fix: Added Misc Chinese models

Pull Request - State: closed - Opened by x-tabdeveloping 22 days ago - 4 comments

#1818 - ci: only return 1 model_name per file

Pull Request - State: closed - Opened by isaac-chung 22 days ago

#1817 - fix: Add additional contacts

Pull Request - State: closed - Opened by KennethEnevoldsen 22 days ago

#1815 - [mieb] Remove results-mieb folder

Pull Request - State: closed - Opened by isaac-chung 22 days ago - 1 comment

#1814 - fix: Added more Chinese models' `ModelMeta`

Pull Request - State: closed - Opened by x-tabdeveloping 22 days ago - 1 comment

#1813 - Leaderboard: What metric should we sort on?

Issue - State: closed - Opened by Samoed 23 days ago - 12 comments
Labels: leaderboard

#1812 - Leaderboard: GritLM-7b score is NaN on STS22.v1

Issue - State: closed - Opened by x-tabdeveloping 23 days ago - 3 comments

#1811 - fix: Added `ModelMeta` for BGE, GTE Chinese and multilingual models

Pull Request - State: closed - Opened by x-tabdeveloping 23 days ago - 1 comment

#1810 - fix: Zero shot and aggregation on Leaderboard

Pull Request - State: closed - Opened by x-tabdeveloping 23 days ago

#1809 - Leaderboard: Mean results scored incorrectly

Issue - State: closed - Opened by Samoed 23 days ago - 4 comments
Labels: leaderboard

#1808 - Leaderboard(Scandinavian) : Missing most results on DKHate

Issue - State: open - Opened by x-tabdeveloping 23 days ago - 6 comments
Labels: bug, leaderboard

#1807 - Leaderboard: Zero-shot filter is buggy

Issue - State: closed - Opened by x-tabdeveloping 23 days ago - 1 comment
Labels: bug, leaderboard

#1805 - fix: Added `ModelMeta` for Chinese and multilingual GTE and BGE models

Pull Request - State: closed - Opened by x-tabdeveloping 23 days ago - 7 comments

#1804 - fix: Fixed eval split for MultilingualSentiment in C-MTEB

Pull Request - State: closed - Opened by x-tabdeveloping 23 days ago - 1 comment

#1803 - Missing `ModelMeta` for Chinese models

Issue - State: closed - Opened by x-tabdeveloping 23 days ago - 11 comments
Labels: good first issue, help wanted, leaderboard

#1802 - docs: Rework MIEB docs

Pull Request - State: closed - Opened by isaac-chung 24 days ago - 5 comments

#1801 - fix: allow to load no revision available

Pull Request - State: closed - Opened by Samoed 24 days ago - 1 comment

#1800 - [MIEB] Complete all model meta based on main schema

Issue - State: closed - Opened by isaac-chung 24 days ago - 2 comments
Labels: mieb

#1799 - [MIEB] Fill out all task description stats metadata

Issue - State: open - Opened by isaac-chung 24 days ago - 2 comments
Labels: mieb

#1798 - fix: loading pre 11

Pull Request - State: closed - Opened by Samoed 24 days ago - 4 comments

#1797 - Leaderboard: Missing results for C-MTEB

Issue - State: open - Opened by x-tabdeveloping 24 days ago - 22 comments

#1796 - docs: Add contact to MMTEB benchmarks

Pull Request - State: closed - Opened by isaac-chung 24 days ago

#1795 - fix: remove default params `TaskMetadata`

Pull Request - State: closed - Opened by Samoed 24 days ago - 1 comment

#1794 - fix: Remove default params, `public_training_data` and `memory usage` in `ModelMeta`

Pull Request - State: closed - Opened by Samoed 24 days ago - 14 comments

#1793 - fix: add even more training dataset annotations

Pull Request - State: closed - Opened by KennethEnevoldsen 24 days ago

#1792 - [MIEB] performance of VOC2007

Issue - State: closed - Opened by gowitheflow-1998 24 days ago - 10 comments
Labels: mieb

#1791 - Merge v2.0.0: Overview issue

Issue - State: open - Opened by KennethEnevoldsen 24 days ago - 2 comments

#1790 - fix: Added contacts to benchmarks

Pull Request - State: closed - Opened by KennethEnevoldsen 24 days ago - 4 comments

#1789 - Add contact person to benchmarks

Issue - State: closed - Opened by KennethEnevoldsen 24 days ago - 6 comments
Labels: enhancement

#1788 - language splits in benchmarks

Issue - State: closed - Opened by KennethEnevoldsen 24 days ago - 1 comment

#1786 - fix: Added C-MTEB

Pull Request - State: closed - Opened by x-tabdeveloping 24 days ago - 10 comments

#1785 - ci: skip AfriSentiLID for now

Pull Request - State: closed - Opened by isaac-chung 25 days ago

#1785 - ci: skip AfriSentiLID for now

Pull Request - State: closed - Opened by isaac-chung 25 days ago

#1784 - [v2] Refactor to Use Enums Instead of Literals

Issue - State: open - Opened by sam-hey 25 days ago - 8 comments
Labels: v2

#1783 - Add C-MTEB to benchmarks.py

Issue - State: open - Opened by x-tabdeveloping 25 days ago - 4 comments

#1783 - Add C-MTEB to benchmarks.py

Issue - State: closed - Opened by x-tabdeveloping 25 days ago - 4 comments

#1782 - Overwrite results when task revision changes

Issue - State: open - Opened by gaohongkui 25 days ago - 1 comment
Labels: enhancement

#1782 - Overwrite results when task revision changes

Issue - State: open - Opened by gaohongkui 25 days ago - 1 comment
Labels: enhancement

#1781 - Integrate Gene-MTEB

Issue - State: open - Opened by Muennighoff 25 days ago
Labels: new benchmark

#1781 - Integrate Gene-MTEB

Issue - State: open - Opened by Muennighoff 25 days ago
Labels: new benchmark

#1780 - [mieb] docs: create task table

Issue - State: closed - Opened by isaac-chung 25 days ago - 5 comments
Labels: mieb

#1780 - [mieb] docs: create task table

Issue - State: closed - Opened by isaac-chung 25 days ago - 5 comments
Labels: mieb

#1779 - [v2] Update `loader` in `ModelMeta` parameters

Issue - State: open - Opened by Samoed 25 days ago - 1 comment
Labels: v2

#1779 - [v2] Updated `ModelMeta` parameters

Issue - State: open - Opened by Samoed 25 days ago - 1 comment

#1778 - [v2] suggestion rename mteb.MTEB to mteb.Evalator

Issue - State: closed - Opened by KennethEnevoldsen 25 days ago - 2 comments

#1778 - [v2] suggestion rename mteb.MTEB to mteb.Evalator

Issue - State: closed - Opened by KennethEnevoldsen 25 days ago - 2 comments

#1777 - Missing HF dataset: HausaNLP/afrisenti-lid-data

Issue - State: open - Opened by isaac-chung 25 days ago - 4 comments

#1777 - Missing HF dataset: HausaNLP/afrisenti-lid-data

Issue - State: open - Opened by isaac-chung 25 days ago - 4 comments

#1776 - Update cross-encoders to use `model_prompts`

Issue - State: open - Opened by orionw 25 days ago

#1775 - ci: fix model loading test

Pull Request - State: closed - Opened by isaac-chung 26 days ago - 2 comments

#1775 - ci: fix model loading test

Pull Request - State: closed - Opened by isaac-chung 26 days ago - 2 comments

#1774 - fix: [v2] _run_eval() for case: co2_tracker False & add test

Pull Request - State: closed - Opened by sam-hey 26 days ago

#1774 - fix: [v2] _run_eval() for case: co2_tracker False & add test

Pull Request - State: closed - Opened by sam-hey 26 days ago

#1773 - [v2] Remove deprecated parameters from `MTEB` and cli

Pull Request - State: closed - Opened by Samoed 26 days ago - 1 comment

#1772 - fix: update max tokens for OpenAI

Pull Request - State: closed - Opened by Samoed 26 days ago

#1772 - fix: update max tokens for OpenAI

Pull Request - State: closed - Opened by Samoed 26 days ago

#1771 - fix: Allow aggregated tasks within benchmarks

Pull Request - State: closed - Opened by KennethEnevoldsen 26 days ago - 3 comments

#1771 - fix: Allow aggregated tasks within benchmarks

Pull Request - State: closed - Opened by KennethEnevoldsen 26 days ago - 3 comments

#1770 - [v2] remove save_suffix from AbsTask

Issue - State: closed - Opened by KennethEnevoldsen 26 days ago - 1 comment
Labels: v2

#1770 - [v2] remove save_suffix from AbsTask

Issue - State: closed - Opened by KennethEnevoldsen 26 days ago - 1 comment
Labels: v2

#1769 - Integrate vdr-2b-multilingual

Issue - State: open - Opened by Muennighoff 26 days ago
Labels: new model, new dataset, mieb

#1768 - feat: add instruct wrapper

Pull Request - State: closed - Opened by Samoed 26 days ago - 6 comments

#1767 - [v2] convert category from length descriptor to modality in task metadata

Issue - State: open - Opened by KennethEnevoldsen 26 days ago - 7 comments
Labels: v2

#1766 - [v2] Getting password prompt for task and tests

Issue - State: open - Opened by sam-hey 26 days ago - 9 comments
Labels: v2

#1766 - [v2] Getting password prompt for task and tests

Issue - State: open - Opened by sam-hey 26 days ago - 9 comments
Labels: v2

#1765 - fix: Added way more training dataset annotations

Pull Request - State: open - Opened by KennethEnevoldsen 26 days ago - 2 comments

#1765 - fix: Added way more training dataset annotations

Pull Request - State: closed - Opened by KennethEnevoldsen 26 days ago - 2 comments

#1764 - Leaderboard: Infinite length models are not shown in "Performance per Model Size"

Issue - State: closed - Opened by KennethEnevoldsen 26 days ago - 2 comments
Labels: leaderboard

#1763 - Missing results for CQDupstack*Retrieval

Issue - State: closed - Opened by KennethEnevoldsen 26 days ago - 1 comment
Labels: leaderboard

#1763 - Missing results for CQDupstack*Retrieval

Issue - State: closed - Opened by KennethEnevoldsen 26 days ago - 1 comment
Labels: leaderboard

#1762 - Leaderboard: Certain models missing in "Performance per Task"

Issue - State: closed - Opened by KennethEnevoldsen 26 days ago
Labels: leaderboard

#1762 - Leaderboard: Certain models missing in "Performance per Task"

Issue - State: closed - Opened by KennethEnevoldsen 26 days ago
Labels: leaderboard