Ecosyste.ms: Issues

An open API service for providing issue and pull request metadata for open source projects.

GitHub / embeddings-benchmark/mteb issues and pull requests

#1761 - fix: Leaderboard: `K` instead of `M`

Pull Request - State: closed - Opened by KennethEnevoldsen 26 days ago

#1761 - fix: Leaderboard: `K` instead of `M`

Pull Request - State: closed - Opened by KennethEnevoldsen 26 days ago

#1760 - Defining zero-shot for MTEB

Issue - State: open - Opened by KennethEnevoldsen 27 days ago - 2 comments
Labels: leaderboard

#1759 - [v2] add similarity_fn in ModelMeta

Pull Request - State: closed - Opened by sam-hey 27 days ago - 24 comments

#1758 - other: add script for leaderboard compare

Pull Request - State: closed - Opened by Samoed 27 days ago

#1758 - other: add script for leaderboard compare

Pull Request - State: closed - Opened by Samoed 27 days ago

#1757 - Benchmark tasks are incorrectly selected

Issue - State: closed - Opened by Samoed 27 days ago - 2 comments

#1756 - Error Model Loading / extract-and-run

Issue - State: closed - Opened by Samoed 27 days ago - 15 comments
Labels: bug

#1756 - Error Model Loading / extract-and-run

Issue - State: closed - Opened by Samoed 27 days ago - 4 comments
Labels: bug

#1755 - Leaderboard: Share this benchmark does not work for custom benchmarks

Issue - State: open - Opened by Muennighoff 27 days ago - 3 comments
Labels: enhancement, leaderboard

#1755 - Leaderboard: Share this benchmark does not work for custom benchmarks

Issue - State: open - Opened by Muennighoff 27 days ago - 3 comments
Labels: enhancement, leaderboard

#1754 - Leaderboard: SFR-Embedding results don't match between old and new

Issue - State: open - Opened by Muennighoff 27 days ago - 4 comments

#1752 - Leaderboard: `K` instead of `M`

Issue - State: closed - Opened by Muennighoff 27 days ago - 1 comment

#1752 - Leaderboard: `K` instead of `M`

Issue - State: closed - Opened by Muennighoff 27 days ago - 1 comment

#1751 - [v2] Remove memory usage

Pull Request - State: closed - Opened by Samoed 27 days ago

#1751 - [v2] Remove memory usage

Pull Request - State: closed - Opened by Samoed 27 days ago

#1750 - [v2] Refactor retrieval

Pull Request - State: closed - Opened by Samoed 27 days ago - 4 comments

#1749 - [v2] fix contriever (add similarity_fn_name to ST wrapper)

Pull Request - State: closed - Opened by Samoed 27 days ago - 4 comments

#1749 - [v2] fix contriever (add similarity_fn_name to ST wrapper)

Pull Request - State: closed - Opened by Samoed 27 days ago - 4 comments

#1748 - fix: fixes implementation of similarity()

Pull Request - State: closed - Opened by sam-hey 27 days ago

#1747 - fix: Fixed definition of zero-shot in ModelMeta

Pull Request - State: closed - Opened by x-tabdeveloping 27 days ago

#1747 - fix: Fixed definition of zero-shot in ModelMeta

Pull Request - State: closed - Opened by x-tabdeveloping 27 days ago

#1746 - fix: Fixed task_type aggregation on leaderboard

Pull Request - State: closed - Opened by x-tabdeveloping 27 days ago

#1746 - fix: Fixed task_type aggregation on leaderboard

Pull Request - State: closed - Opened by x-tabdeveloping 27 days ago

#1745 - fix: Leaderboard Speedup

Pull Request - State: closed - Opened by x-tabdeveloping 28 days ago

#1745 - fix: Leaderboard Speedup

Pull Request - State: closed - Opened by x-tabdeveloping 28 days ago

#1744 - Add model inf-retriever-v1

Pull Request - State: closed - Opened by SamuelYang1 28 days ago - 2 comments

#1744 - Add model inf-retriever-v1

Pull Request - State: open - Opened by SamuelYang1 28 days ago - 2 comments

#1743 - fix: change comparison to bigger than

Pull Request - State: closed - Opened by isaac-chung 28 days ago

#1742 - fix: added annotations for training data

Pull Request - State: closed - Opened by KennethEnevoldsen 28 days ago - 4 comments

#1742 - fix: added annotations for training data

Pull Request - State: closed - Opened by KennethEnevoldsen 28 days ago - 4 comments

#1741 - Add giga embeddings

Pull Request - State: open - Opened by Samoed 29 days ago

#1741 - Add giga embeddings

Pull Request - State: open - Opened by Samoed 29 days ago

#1740 - Leaderboard: Speed up `get_scores()`

Issue - State: closed - Opened by x-tabdeveloping 29 days ago - 3 comments

#1740 - Leaderboard: Speed up `get_scores()`

Issue - State: closed - Opened by x-tabdeveloping 29 days ago - 3 comments

#1739 - Fixed result loading on leaderboard

Pull Request - State: closed - Opened by x-tabdeveloping 29 days ago - 1 comment

#1738 - downsample MIRACLReranking

Issue - State: open - Opened by KennethEnevoldsen 29 days ago
Labels: enhancement, good first issue

#1737 - Downsample MIRACLReranking using MIRACLRetrievalHardNegatives

Issue - State: open - Opened by KennethEnevoldsen 29 days ago
Labels: enhancement, good first issue

#1737 - Downsample MIRACLReranking using MIRACLRetrievalHardNegatives

Issue - State: open - Opened by KennethEnevoldsen 29 days ago
Labels: enhancement, good first issue

#1736 - fix: Fix zeta alpha mistral

Pull Request - State: closed - Opened by Samoed 29 days ago - 6 comments

#1734 - Discrepancy in Cross-Encoder Scores Between MTEB and SentenceTransformers

Issue - State: closed - Opened by sam-hey 29 days ago - 10 comments

#1734 - Discrepancy in Cross-Encoder Scores Between MTEB and SentenceTransformers

Issue - State: closed - Opened by sam-hey 29 days ago - 10 comments

#1733 - Fix RepLLaMA-based models and Instructions for Cross-Encoders

Pull Request - State: closed - Opened by orionw 29 days ago - 9 comments

#1733 - Fix RepLLaMA-based models and Instructions for Cross-Encoders

Pull Request - State: closed - Opened by orionw 29 days ago - 9 comments

#1732 - Potentially relevant multi turn rag dataset

Issue - State: open - Opened by KennethEnevoldsen 29 days ago
Labels: new dataset

#1732 - Potentially relevant multi turn rag dataset

Issue - State: open - Opened by KennethEnevoldsen 29 days ago
Labels: new dataset

#1730 - feat: reduce logging for load_results()

Pull Request - State: closed - Opened by KennethEnevoldsen 29 days ago - 2 comments

#1730 - feat: reduce logging for load_results()

Pull Request - State: closed - Opened by KennethEnevoldsen 29 days ago - 2 comments

#1729 - Remove "memory_usage" from ModelMeta

Issue - State: closed - Opened by KennethEnevoldsen 29 days ago

#1729 - Remove "memory_usage" from ModelMeta

Issue - State: closed - Opened by KennethEnevoldsen 29 days ago

#1728 - Add CLEF to MTEB

Issue - State: closed - Opened by KennethEnevoldsen 29 days ago - 2 comments

#1728 - Add CLEF to MTEB

Issue - State: closed - Opened by KennethEnevoldsen 29 days ago - 2 comments

#1727 - How should we handle gated models when testing

Issue - State: open - Opened by KennethEnevoldsen 29 days ago - 2 comments

#1727 - How should we handle gated models when testing

Issue - State: open - Opened by KennethEnevoldsen 29 days ago - 2 comments

#1725 - Add support for STS evaluation for Cross-Encoders

Issue - State: closed - Opened by sam-hey 29 days ago - 8 comments
Labels: v2

#1725 - Add support for STS evaluation for Cross-Encoders

Issue - State: closed - Opened by sam-hey 29 days ago - 8 comments
Labels: v2

#1724 - Leaderboard demo down: Results are too straining to load

Issue - State: closed - Opened by x-tabdeveloping 29 days ago - 6 comments

#1724 - Leaderboard demo down: Results are too straining to load

Issue - State: closed - Opened by x-tabdeveloping 29 days ago - 6 comments

#1723 - fix: add version check e5-v in mieb

Pull Request - State: closed - Opened by isaac-chung 30 days ago

#1723 - fix: add version check e5-v in mieb

Pull Request - State: closed - Opened by isaac-chung 30 days ago

#1721 - How to calculate MTEB score from individual files?

Issue - State: open - Opened by orionw about 1 month ago - 2 comments

#1721 - How to calculate MTEB score from individual files?

Issue - State: open - Opened by orionw about 1 month ago - 2 comments

#1720 - Leaderboard: Add training data annotations for models

Issue - State: closed - Opened by x-tabdeveloping about 1 month ago - 3 comments
Labels: good first issue, leaderboard

#1718 - Inconsistent casing for MIRACL metrics

Issue - State: open - Opened by Muennighoff about 1 month ago - 1 comment

#1717 - malenia1/ternary-weight-embedding does not load weights

Issue - State: open - Opened by Muennighoff about 1 month ago

#1715 - Add test to ensure that the leaderboard can build

Issue - State: open - Opened by KennethEnevoldsen about 1 month ago
Labels: leaderboard

#1715 - Add test to ensure that the leaderboard can build

Issue - State: open - Opened by KennethEnevoldsen about 1 month ago
Labels: leaderboard

#1713 - Remove `metadata_dict`

Issue - State: open - Opened by KennethEnevoldsen about 1 month ago

#1713 - Remove `metadata_dict`

Issue - State: open - Opened by KennethEnevoldsen about 1 month ago

#1712 - integrate VSI-BENCH

Issue - State: open - Opened by Samoed about 1 month ago
Labels: modality: image, new benchmark

#1712 - integrate VSI-BENCH

Issue - State: open - Opened by Samoed about 1 month ago
Labels: modality: image, new benchmark

#1711 - Refresh the v2 leaderboard daily

Pull Request - State: closed - Opened by orionw about 1 month ago - 1 comment

#1711 - Refresh the v2 leaderboard daily

Pull Request - State: closed - Opened by orionw about 1 month ago - 1 comment

#1710 - Zero shot

Pull Request - State: closed - Opened by x-tabdeveloping about 1 month ago - 10 comments

#1709 - Add MMEB

Issue - State: open - Opened by Muennighoff about 1 month ago
Labels: modality: image, new benchmark

#1709 - Add MMEB

Issue - State: open - Opened by Muennighoff about 1 month ago
Labels: modality: image, new benchmark

#1708 - feat: Integrating ChemTEB

Pull Request - State: closed - Opened by HSILA about 1 month ago - 8 comments

#1708 - feat: Integrating ChemTEB

Pull Request - State: open - Opened by HSILA about 1 month ago - 7 comments

#1707 - [v2] Refactor evaluators and Abstasks

Pull Request - State: closed - Opened by Samoed about 1 month ago - 7 comments

#1706 - fix: bucc

Pull Request - State: open - Opened by Samoed about 1 month ago

#1706 - fix: bucc

Pull Request - State: open - Opened by Samoed about 1 month ago

#1705 - Fix: minicpmv2

Pull Request - State: closed - Opened by Samoed about 1 month ago

#1705 - Fix: minicpmv2

Pull Request - State: closed - Opened by Samoed about 1 month ago

#1704 - fix: bucc

Pull Request - State: closed - Opened by Samoed about 1 month ago

#1704 - fix: bucc

Pull Request - State: closed - Opened by Samoed about 1 month ago

#1703 - fix: GermanDPR

Pull Request - State: closed - Opened by Samoed about 1 month ago - 1 comment

#1703 - fix: GermanDPR

Pull Request - State: closed - Opened by Samoed about 1 month ago - 1 comment

#1702 - Update RerankingEvaluator.py

Pull Request - State: closed - Opened by Muennighoff about 1 month ago

#1702 - Update RerankingEvaluator.py

Pull Request - State: closed - Opened by Muennighoff about 1 month ago

#1701 - May I ask if MTEB supports evaluating downloaded data？

Issue - State: open - Opened by BtlWolf about 1 month ago - 4 comments

#1700 - GermanDPR fails

Issue - State: closed - Opened by Muennighoff about 1 month ago

#1700 - GermanDPR fails

Issue - State: closed - Opened by Muennighoff about 1 month ago

#1699 - Add benchmark on word order

Issue - State: open - Opened by KennethEnevoldsen about 1 month ago
Labels: enhancement, new benchmark

#1698 - test: Add script to test model loading below n_parameters threshold

Pull Request - State: closed - Opened by isaac-chung about 1 month ago - 8 comments