Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / embeddings-benchmark/mteb issues and pull requests

#1641 - add missing benchmark to benchmarks.py

Pull Request - State: closed - Opened by gowitheflow-1998 about 1 month ago

#1640 - [mieb] model task modalities matching logic

Pull Request - State: closed - Opened by gowitheflow-1998 about 1 month ago

#1639 - Warn

Pull Request - State: closed - Opened by Muennighoff about 1 month ago

#1639 - Warn

Pull Request - State: closed - Opened by Muennighoff about 1 month ago

#1638 - [MIEB] Fix VISTA model

Pull Request - State: closed - Opened by izhx about 1 month ago

#1637 - Correction of discrepancies for gte-Qweb model

Pull Request - State: closed - Opened by AlexeyVatolin about 1 month ago - 2 comments

#1636 - autodetect dataset contamination

Issue - State: open - Opened by KennethEnevoldsen about 1 month ago
Labels: leaderboard

#1635 - [MIEB] Add code for GME models

Pull Request - State: closed - Opened by izhx about 1 month ago

#1634 - Annotate models trained on commercial-use-friendly licensed data

Issue - State: open - Opened by nramrakhiyani about 1 month ago - 5 comments
Labels: enhancement, leaderboard

#1634 - Annotate models trained on commercial-use-friendly licensed data

Issue - State: open - Opened by nramrakhiyani about 1 month ago - 5 comments
Labels: enhancement, leaderboard

#1633 - [mieb] Task filtering by modality supported by models

Pull Request - State: closed - Opened by isaac-chung about 1 month ago - 3 comments

#1633 - [mieb] Task filtering by modality supported by models

Pull Request - State: closed - Opened by isaac-chung about 1 month ago - 3 comments

#1632 - [mieb] Error message for unsupported models is unclear

Issue - State: closed - Opened by Muennighoff about 1 month ago - 2 comments
Labels: mieb

#1631 - Speed up test_save_predictions

Pull Request - State: closed - Opened by AlexeyVatolin about 1 month ago

#1630 - Speed up test_save_predictions

Issue - State: closed - Opened by isaac-chung about 1 month ago

#1630 - Speed up test_save_predictions

Issue - State: closed - Opened by isaac-chung about 1 month ago

#1629 - Extend MIEB test coverage

Pull Request - State: closed - Opened by isaac-chung about 1 month ago

#1629 - Extend MIEB test coverage

Pull Request - State: closed - Opened by isaac-chung about 1 month ago

#1628 - Update results_to_dataframe to use BenchmarkResults class

Pull Request - State: closed - Opened by AlexeyVatolin about 1 month ago

#1628 - Update results_to_dataframe to use BenchmarkResults class

Pull Request - State: closed - Opened by AlexeyVatolin about 1 month ago

#1627 - Abnormally low values for NanoBEIR benchmark

Issue - State: closed - Opened by minsik-ai about 1 month ago - 10 comments
Labels: bug

#1626 - added local_path in task metadata_dict for loading local dataset

Pull Request - State: closed - Opened by ashokrajab about 1 month ago

#1626 - added local_path in task metadata_dict for loading local dataset

Pull Request - State: closed - Opened by ashokrajab about 1 month ago

#1624 - Integrate ModernBERT

Issue - State: closed - Opened by Samoed about 2 months ago - 4 comments
Labels: new model

#1623 - fix: properly add mteb_model_meta to model object

Pull Request - State: closed - Opened by KennethEnevoldsen about 2 months ago

#1622 - Leaderboard: Add ONNX and GGUF to frameworks to allow us to filter based on these

Issue - State: open - Opened by KennethEnevoldsen about 2 months ago - 1 comment
Labels: leaderboard

#1620 - add MSMARCO eval split in MTEB English (classic) benchmark

Pull Request - State: closed - Opened by KennethEnevoldsen about 2 months ago

#1619 - [V2] Update datasets wich can't be loaded with `datasets>=3.0`

Pull Request - State: closed - Opened by Samoed about 2 months ago

#1618 - [V2] Update v2

Pull Request - State: closed - Opened by Samoed about 2 months ago - 3 comments

#1617 - fix: override existing results

Pull Request - State: closed - Opened by Samoed about 2 months ago - 2 comments

#1616 - fix: set `use_instructions` to True in models using prompts

Pull Request - State: closed - Opened by dbuades about 2 months ago - 1 comment

#1614 - fix: disable co2_tracker for API models

Pull Request - State: closed - Opened by dbuades about 2 months ago - 1 comment

#1613 - Add IBM Granite Embedding Models

Pull Request - State: closed - Opened by aashka-trivedi about 2 months ago

#1612 - [MIEB] Fix `get_fused_emebddings`

Pull Request - State: closed - Opened by izhx about 2 months ago - 3 comments

#1611 - [MIEB] Add new multimodal retrieval tasks

Pull Request - State: closed - Opened by izhx about 2 months ago

#1610 - [mieb] voyage-v: add exponential backoff and other error handling

Pull Request - State: closed - Opened by gowitheflow-1998 about 2 months ago - 1 comment

#1609 - GermanDPR Dataset Causes Cross-Encoder Failure Due to Unexpected dict

Issue - State: closed - Opened by sam-hey about 2 months ago - 3 comments
Labels: bug

#1608 - MSMARCO eval split in MTEB English (classic) benchmark

Issue - State: closed - Opened by aashka-trivedi about 2 months ago - 4 comments
Labels: bug

#1607 - [MIEB] Switch to ViDoRe BEIR version

Pull Request - State: closed - Opened by izhx about 2 months ago - 4 comments

#1606 - Normalize image encoders interface to the text encoder interface

Issue - State: open - Opened by KennethEnevoldsen about 2 months ago - 1 comment

#1605 - leaderboard v2 - add autoupdate

Issue - State: closed - Opened by KennethEnevoldsen about 2 months ago - 5 comments
Labels: leaderboard

#1603 - fix: Leaderboard refinements

Pull Request - State: closed - Opened by x-tabdeveloping about 2 months ago

#1603 - fix: Leaderboard refinements

Pull Request - State: closed - Opened by x-tabdeveloping about 2 months ago

#1602 - Feat: Use similarity scores if available

Pull Request - State: closed - Opened by Samoed about 2 months ago

#1601 - Leaderboard: Refined plots

Pull Request - State: closed - Opened by x-tabdeveloping about 2 months ago

#1601 - Leaderboard: Refined plots

Pull Request - State: closed - Opened by x-tabdeveloping about 2 months ago

#1600 - Investigate performance discrepancies in gte-Qwen and NV-embed models

Issue - State: open - Opened by isaac-chung about 2 months ago - 10 comments

#1597 - fix: reorder argument for mteb.get_tasks

Pull Request - State: open - Opened by KennethEnevoldsen about 2 months ago - 1 comment

#1597 - fix: reorder argument for mteb.get_tasks

Pull Request - State: closed - Opened by KennethEnevoldsen about 2 months ago - 1 comment

#1596 - fix image encoder

Pull Request - State: closed - Opened by KennethEnevoldsen about 2 months ago

#1596 - fix image encoder

Pull Request - State: closed - Opened by KennethEnevoldsen about 2 months ago

#1595 - fix: Add datasets in CodeRAG-Bench

Pull Request - State: closed - Opened by hepengfe about 2 months ago - 10 comments

#1595 - fix: Add datasets in CodeRAG-Bench

Pull Request - State: closed - Opened by hepengfe about 2 months ago - 10 comments

#1594 - Feat: add support for scoring function

Pull Request - State: closed - Opened by Samoed about 2 months ago - 7 comments

#1593 - Add Support for Index-Based Retrieval for ColBERT

Issue - State: open - Opened by sam-hey about 2 months ago
Labels: enhancement

#1593 - Add Support for Index-Based Retrieval for ColBERT

Issue - State: open - Opened by sam-hey about 2 months ago
Labels: enhancement

#1592 - doc: colbert add score_function & doc section

Pull Request - State: closed - Opened by sam-hey about 2 months ago

#1591 - Feat: Add jasper

Pull Request - State: closed - Opened by Samoed about 2 months ago - 14 comments

#1590 - Integrate UniBench

Issue - State: open - Opened by Samoed about 2 months ago - 20 comments
Labels: mieb, new benchmark

#1590 - Integrate UniBench

Issue - State: open - Opened by Samoed about 2 months ago
Labels: new-dataset, mieb

#1588 - Add NanoBEIR Datasets

Pull Request - State: open - Opened by KGupta10 about 2 months ago - 2 comments

#1588 - Add NanoBEIR Datasets

Pull Request - State: closed - Opened by KGupta10 about 2 months ago - 5 comments

#1587 - fix: Eval langs not correctly passed to monolingual tasks

Pull Request - State: closed - Opened by Samoed about 2 months ago - 1 comment

#1585 - Integrate ChemTEB

Issue - State: closed - Opened by Muennighoff about 2 months ago - 7 comments
Labels: new benchmark

#1585 - Integrate ChemTEB

Issue - State: open - Opened by Muennighoff about 2 months ago - 1 comment

#1584 - Feat: Evaluate missing languages

Pull Request - State: closed - Opened by Samoed about 2 months ago - 3 comments

#1584 - Feat: Evaluate missing languages

Pull Request - State: open - Opened by Samoed about 2 months ago - 3 comments

#1583 - [MIEB] Make multimodal models compatible to `task_name` and `prompt_type`

Pull Request - State: closed - Opened by izhx about 2 months ago - 3 comments

#1583 - [MIEB] Make multimodal models compatible to `task_name` and `prompt_type`

Pull Request - State: closed - Opened by izhx about 2 months ago - 3 comments

#1582 - [MIEB] Update VLM2Vec model

Issue - State: open - Opened by XMHZZ2018 about 2 months ago - 1 comment

#1582 - [MIEB] Update VLM2Vec model

Issue - State: open - Opened by XMHZZ2018 about 2 months ago - 1 comment
Labels: mieb

#1580 - leaderboard v2: it is unclear what the size of the circle is

Issue - State: closed - Opened by KennethEnevoldsen about 2 months ago - 4 comments
Labels: leaderboard

#1579 - We should be able to filter GGUF models

Issue - State: closed - Opened by KennethEnevoldsen about 2 months ago - 5 comments
Labels: enhancement, leaderboard

#1578 - Update

Pull Request - State: closed - Opened by Muennighoff about 2 months ago

#1578 - Update

Pull Request - State: closed - Opened by Muennighoff about 2 months ago

#1576 - [mieb] update script for final re-run

Pull Request - State: closed - Opened by gowitheflow-1998 about 2 months ago - 1 comment

#1575 - Merge main to v2 again

Pull Request - State: closed - Opened by Samoed about 2 months ago

#1575 - Merge main to v2 again

Pull Request - State: closed - Opened by Samoed about 2 months ago

#1574 - feat: add new arctic v2.0 models

Pull Request - State: closed - Opened by dbuades about 2 months ago - 3 comments

#1574 - feat: add new arctic v2.0 models

Pull Request - State: closed - Opened by dbuades about 2 months ago - 3 comments

#1573 - fix: Add namaa MrTydi reranking dataset

Pull Request - State: closed - Opened by omarelshehy about 2 months ago - 3 comments

#1573 - fix: Add namaa MrTydi reranking dataset

Pull Request - State: closed - Opened by omarelshehy about 2 months ago - 3 comments

#1572 - fix: no longer using same query text for all of BLINKIT2TMultiChoice

Pull Request - State: closed - Opened by Jamie-Stirling about 2 months ago - 1 comment

#1572 - fix: no longer using same query text for all of BLINKIT2TMultiChoice

Pull Request - State: closed - Opened by Jamie-Stirling about 2 months ago - 1 comment

#1571 - Leaderboard 2.0: Missing results

Issue - State: closed - Opened by x-tabdeveloping about 2 months ago - 18 comments
Labels: leaderboard

#1570 - fix: Added radar chart displaying capabilities on task types

Pull Request - State: closed - Opened by x-tabdeveloping about 2 months ago - 1 comment

#1570 - fix: Added radar chart displaying capabilities on task types

Pull Request - State: closed - Opened by x-tabdeveloping about 2 months ago - 1 comment

#1569 - fix: remove `*` imports

Pull Request - State: closed - Opened by Samoed 2 months ago - 4 comments