Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / embeddings-benchmark/mteb issues and pull requests

#1949 - Fixed join_revisions if results are empty

Pull Request - State: closed - Opened by x-tabdeveloping 2 days ago

#1948 - Updated links in MTEB(eng) and eng,classic

Pull Request - State: closed - Opened by x-tabdeveloping 3 days ago

#1946 - [v2] Integrate cde models

Issue - State: open - Opened by Samoed 3 days ago
Labels: v2

#1945 - misc: add warnings

Pull Request - State: open - Opened by Samoed 3 days ago - 1 comment

#1944 - feat: Merge MIEB into main 🎉

Pull Request - State: closed - Opened by isaac-chung 3 days ago - 2 comments

#1943 - Evaluation on ArxivClassfiication: Input X contains NaN.

Issue - State: open - Opened by JunyiZhu-AI 3 days ago - 2 comments

#1942 - [mieb] merge from main once more

Pull Request - State: closed - Opened by isaac-chung 4 days ago - 1 comment

#1940 - misc: add warning for save_suffix removal from AbsTask

Pull Request - State: closed - Opened by isaac-chung 4 days ago

#1938 - Github Action bot producting a lot of commits

Issue - State: closed - Opened by Samoed 4 days ago - 1 comment

#1937 - Integrate SuperGLUE

Issue - State: open - Opened by Samoed 4 days ago
Labels: new benchmark

#1936 - [v2] Reupload `BEIR-PL`

Issue - State: open - Opened by Samoed 4 days ago
Labels: v2

#1935 - [Leaderboard] Autocalculate memory usage in model meta

Issue - State: open - Opened by Samoed 4 days ago - 2 comments
Labels: enhancement, leaderboard

#1934 - feat: add model memory usage

Pull Request - State: open - Opened by Samoed 4 days ago - 2 comments

#1933 - feat: add beir

Pull Request - State: closed - Opened by Samoed 4 days ago

#1932 - Fix training data for gritlm

Pull Request - State: open - Opened by Muennighoff 4 days ago

#1930 - Leaderboard: Acks

Pull Request - State: closed - Opened by Muennighoff 4 days ago - 2 comments

#1929 - [v2] Reupload datasets with `fast_loading`

Issue - State: open - Opened by Samoed 4 days ago
Labels: v2

#1928 - misc: add bgev1 models

Pull Request - State: closed - Opened by Samoed 4 days ago

#1927 - [v2] Merge main

Pull Request - State: open - Opened by Samoed 4 days ago - 3 comments
Labels: v2

#1926 - [v2] Remove multilingual task

Pull Request - State: closed - Opened by Samoed 4 days ago - 10 comments
Labels: v2

#1925 - docs: Add instruction for running leaderboard

Pull Request - State: open - Opened by KennethEnevoldsen 4 days ago - 4 comments

#1924 - [v2] remove `save_suffix`

Pull Request - State: closed - Opened by Samoed 5 days ago
Labels: v2

#1922 - docs: Add sort to domains for task metadata

Pull Request - State: closed - Opened by KennethEnevoldsen 5 days ago - 1 comment

#1921 - Leaderboard - Models with missing tasks getting ranked

Issue - State: closed - Opened by Muennighoff 5 days ago - 2 comments

#1920 - Leaderboard: Link for text-embedding-005

Issue - State: open - Opened by Muennighoff 5 days ago

#1919 - French leaderboard inconsistencies

Issue - State: open - Opened by KennethEnevoldsen 5 days ago - 1 comment
Labels: leaderboard

#1918 - revert rename and add to description

Pull Request - State: closed - Opened by isaac-chung 5 days ago

#1917 - Polish leaderboard and benchmark does not match

Issue - State: open - Opened by KennethEnevoldsen 5 days ago
Labels: leaderboard

#1916 - CI: Update task table generated too many commits

Issue - State: closed - Opened by isaac-chung 5 days ago

#1915 - fix: remove SummaryRetrieval as a type

Pull Request - State: closed - Opened by KennethEnevoldsen 5 days ago - 2 comments

#1914 - docs: Updated citation for mteb(scandinavian)

Pull Request - State: closed - Opened by KennethEnevoldsen 5 days ago

#1913 - [Leaderboard] Can't select `FaMTEB`

Issue - State: closed - Opened by Samoed 6 days ago
Labels: leaderboard

#1912 - Discrepancy in Benchmark Results for BAAI/bge-base-en, BAAI/bge-large-en, BAAI/bge-small-en

Issue - State: closed - Opened by noworneverev 6 days ago - 7 comments
Labels: replication

#1911 - Add new clustering tasks for a new domain (Engineering)

Pull Request - State: open - Opened by mehrzadshm 7 days ago - 3 comments

#1910 - fix: Add annotation for cqadupstack

Pull Request - State: open - Opened by KennethEnevoldsen 7 days ago - 1 comment

#1909 - feat: Add new benchmark BEIR-NL

Pull Request - State: open - Opened by nikolay-banar 7 days ago - 3 comments

#1908 - Adding a banner to the new MMTEB leaderboard

Pull Request - State: closed - Opened by wissam-sib 8 days ago - 3 comments

#1907 - feat: Custom search interface

Pull Request - State: closed - Opened by sam-hey 8 days ago - 5 comments

#1906 - Add mini-gte model

Pull Request - State: open - Opened by Ressnn 8 days ago

#1905 - test merge

Pull Request - State: closed - Opened by KennethEnevoldsen 8 days ago

#1904 - Add model spice

Pull Request - State: closed - Opened by iamgroot42 8 days ago - 1 comment

#1903 - [mieb] Fill in vista vlm2vec voyage v model meta

Pull Request - State: closed - Opened by isaac-chung 8 days ago

#1902 - fix: Limited plotly version to be less than 6.0.0

Pull Request - State: closed - Opened by x-tabdeveloping 8 days ago

#1901 - Leaderboard: Main figure doesn't work

Issue - State: closed - Opened by x-tabdeveloping 8 days ago
Labels: leaderboard

#1900 - CQADupstack results are still missing for some models

Issue - State: closed - Opened by x-tabdeveloping 8 days ago - 4 comments

#1899 - Old results cannot be loaded after being saved

Issue - State: open - Opened by x-tabdeveloping 8 days ago - 2 comments

#1898 - Run MSMARCO dev split on some models

Issue - State: open - Opened by x-tabdeveloping 9 days ago - 1 comment

#1896 - update stella/jasper metainfo

Pull Request - State: closed - Opened by Samoed 9 days ago

#1895 - fix: Filling missing metadata for leaderboard release

Pull Request - State: closed - Opened by imenelydiaker 9 days ago - 5 comments

#1891 - Model meta `use_instructions` change

Issue - State: open - Opened by Samoed 9 days ago

#1890 - [mieb] Fill in mocov3 and nomic vision model meta

Pull Request - State: open - Opened by isaac-chung 10 days ago

#1889 - [v2] Fill dataset Metadata

Issue - State: open - Opened by Samoed 10 days ago
Labels: v2

#1888 - fix: update voyage exp metadata

Pull Request - State: closed - Opened by KennethEnevoldsen 10 days ago

#1887 - [mieb] Fill out gme v and jina clip model meta

Pull Request - State: closed - Opened by isaac-chung 10 days ago

#1886 - Missing Metadata for tasks in `MTEB(eng, classic)`

Issue - State: closed - Opened by x-tabdeveloping 10 days ago - 3 comments
Labels: leaderboard

#1885 - [mieb] Fill in e5v and eva clip model meta

Pull Request - State: closed - Opened by isaac-chung 10 days ago

#1884 - Add model spice

Pull Request - State: closed - Opened by iamgroot42 10 days ago - 6 comments

#1883 - Add GenBench

Issue - State: open - Opened by Samoed 10 days ago
Labels: new benchmark

#1882 - feat: Added Hindi benchmark.

Pull Request - State: open - Opened by SaileshP97 10 days ago

#1881 - Added Hindi benchmark.

Pull Request - State: closed - Opened by SaileshP97 10 days ago - 1 comment

#1880 - [mieb] Fill in cohere_v and dinov2 model meta

Pull Request - State: closed - Opened by isaac-chung 11 days ago

#1879 - Leaderboard: Some models are missing in `MTEB(rus)`

Issue - State: closed - Opened by Samoed 11 days ago - 2 comments
Labels: leaderboard

#1878 - [v2] Reupload Reranking datasets in old format

Issue - State: open - Opened by Samoed 11 days ago
Labels: v2

#1877 - [v2] Reupload datasets with `trust_remote_code`

Issue - State: open - Opened by Samoed 11 days ago
Labels: v2

#1876 - [mieb] Fill in clip and open clip model meta

Pull Request - State: closed - Opened by isaac-chung 11 days ago

#1875 - initialize an arabic mteb benchmark

Pull Request - State: open - Opened by haytamdon 11 days ago

#1874 - [mieb] Fill in blip model meta

Pull Request - State: closed - Opened by isaac-chung 11 days ago

#1873 - Dataset/update climate fever

Pull Request - State: open - Opened by mina-parham 12 days ago - 3 comments

#1872 - fix: fix jina v1, 2 models

Pull Request - State: closed - Opened by Samoed 12 days ago

#1871 - doc: update pr template

Pull Request - State: closed - Opened by sam-hey 12 days ago

#1870 - Docs: update docs according to current state

Pull Request - State: closed - Opened by Samoed 13 days ago - 3 comments

#1869 - add is_cross_encoder

Pull Request - State: open - Opened by Samoed 13 days ago - 7 comments

#1868 - Update docs to cover new model result submission

Issue - State: closed - Opened by isaac-chung 13 days ago - 2 comments

#1867 - Overview: Leaderboard release

Issue - State: open - Opened by x-tabdeveloping 13 days ago - 26 comments
Labels: leaderboard

#1865 - Question about encode function

Issue - State: closed - Opened by yeliu918 14 days ago - 7 comments

#1864 - [v2] Update v2 again

Pull Request - State: closed - Opened by Samoed 14 days ago - 3 comments

#1863 - [mieb] Fill in align model meta

Pull Request - State: closed - Opened by isaac-chung 14 days ago

#1862 - Discrepancy in MTEB Benchmark Results for jinaai/jina-embeddings-v2-small-en

Issue - State: closed - Opened by noworneverev 15 days ago - 4 comments
Labels: replication

#1861 - Discrepencies in CoIR results

Issue - State: open - Opened by bclavie 15 days ago - 6 comments
Labels: replication

#1860 - MMTEB is accepted by ICLR 2025

Issue - State: open - Opened by gentaiscool 15 days ago - 2 comments

#1859 - fix: Added correct training data annotation to LENS

Pull Request - State: closed - Opened by x-tabdeveloping 15 days ago

#1858 - Allow overwrites of pre-filtering

Issue - State: closed - Opened by SaileshP97 16 days ago - 2 comments

#1857 - fix: Hotfixed public_training_data type annotation

Pull Request - State: closed - Opened by x-tabdeveloping 16 days ago - 2 comments

#1856 - fix: Adding missing model meta

Pull Request - State: closed - Opened by x-tabdeveloping 16 days ago - 2 comments

#1855 - New leaderboard banner

Issue - State: closed - Opened by Muennighoff 16 days ago - 3 comments
Labels: leaderboard

#1854 - Information on metrics gone in new leaderboard

Issue - State: closed - Opened by Muennighoff 16 days ago - 3 comments

#1853 - [mieb] Merge from main

Pull Request - State: closed - Opened by isaac-chung 16 days ago - 5 comments

#1852 - fix: Fixed leaderboard search bar

Pull Request - State: closed - Opened by x-tabdeveloping 16 days ago

#1851 - CDE models missing

Issue - State: closed - Opened by x-tabdeveloping 16 days ago - 17 comments

#1850 - Search bar doesn't work on the leaderboard

Issue - State: closed - Opened by x-tabdeveloping 16 days ago - 1 comment
Labels: bug, leaderboard

#1849 - fix: Leaderboard Refinements

Pull Request - State: closed - Opened by x-tabdeveloping 16 days ago - 2 comments

#1848 - Missing models on leaderboards [WIP]

Issue - State: closed - Opened by KennethEnevoldsen 17 days ago - 3 comments
Labels: leaderboard

#1847 - Renaming the New Leaderboard Space

Issue - State: open - Opened by x-tabdeveloping 17 days ago - 8 comments
Labels: leaderboard