beir-cellar/beir issues and pull requests

#187 - Incorrect Evaluation Metrics Produced

Issue - State: open - Opened by NikhielRahulSingh 29 days ago - 2 comments

#186 - 你们的IDCG计算是什么方式

Issue - State: open - Opened by wangjitoan about 2 months ago

#185 - evaluate_sbert_multi_gpu run error

Issue - State: open - Opened by ET-yzk 2 months ago

#184 - BUG - `EvaluateRetrieval`'s `evaluate` doesn't consider results dict order

Issue - State: open - Opened by vd-omermazig 3 months ago

#183 - Sort qrel and results so the keys lexical order fits the dict order.

Pull Request - State: open - Opened by vd-omermazig 3 months ago

#182 - BeIR adaptation for Russian language

Pull Request - State: closed - Opened by kngrg 3 months ago

#181 - how was the hotpot_qa dataset preprocessed?

Issue - State: open - Opened by DanielSchuhmacher 3 months ago

#180 - Training CrossEncoder using BEIR

Issue - State: closed - Opened by jmnian 4 months ago - 2 comments

#179 - NQ - File datasets/nq/qrels/train.tsv not present

Issue - State: open - Opened by GiacoL 4 months ago - 6 comments

#178 - fix: use np.int64 and np.float64 instead of deprecated np.int and np.float

Pull Request - State: open - Opened by DobryninVyacheslav 4 months ago - 1 comment

#177 - Fix: mention proper metric used for retrieval

Pull Request - State: open - Opened by kacperlukawski 5 months ago

#176 - can't reproduce DeepCT results with evaluate_deepct.py

Issue - State: open - Opened by jeka-e 6 months ago

#175 - replacing `pytrec_eval` with `pytrec-eval-terrier`

Pull Request - State: open - Opened by archersama 6 months ago

#174 - recommended pytrec-eval-terrier rather than pytrec-eval

Issue - State: open - Opened by liyongkang123 6 months ago

#173 - mt5-base model (unicamp-dl/mt5-base-en-msmarco) throws 'nan' outputs

Issue - State: open - Opened by cramraj8 6 months ago

#172 - Missing document from corpus ArguAna

Issue - State: open - Opened by NourOM02 7 months ago

#171 - How to Save Retrieved Documents in Retrieval Tasks

Issue - State: open - Opened by AugustLHHHHHH 7 months ago

#170 - `retriever.retrieve(corpus, queries)` with `len(queries)==1` errors

Issue - State: open - Opened by manestay 8 months ago - 1 comment

#169 - Why need check corpus_id != query_id in DenseRetrievalExactSearch.search()

Issue - State: open - Opened by mengyao00 8 months ago - 1 comment

#168 - TypeError: Cannot create a consistent method resolution order (MRO) for bases tqdm, tqdm_asyncio

Issue - State: closed - Opened by dinuransika 8 months ago

#167 - Evaluate "Offline" Models

Pull Request - State: open - Opened by arendu 9 months ago

#166 - Rename Accuracy@k Metric to Success@k metric

Issue - State: open - Opened by root-goksenin 9 months ago

#165 - Corpus encoding times for hotpotqa on A100 GPU

Issue - State: open - Opened by jeyendranbalakrishnan 9 months ago

#164 - adding support for decoder-only LLMs e.g. E5-Mistral

Issue - State: closed - Opened by Krilecy 9 months ago - 2 comments

#163 - example file, train_msmarco_v2.py is not working

Issue - State: open - Opened by bakingeol 10 months ago

#162 - Codespace ubiquitous acorn qpgj6564q7366vj

Pull Request - State: closed - Opened by konabuta 10 months ago

#161 - Malformed Chinese mMARCO dataset

Issue - State: open - Opened by sissilab 11 months ago

#160 - Extras-require, base package without Faiss-CPU requirement

Issue - State: open - Opened by raphaelsty about 1 year ago - 1 comment

#159 - mMarco version

Issue - State: open - Opened by carlesoctav about 1 year ago

#158 - FIQA Empty Rows

Issue - State: closed - Opened by zanussbaum about 1 year ago - 3 comments

#157 - Installation

Issue - State: open - Opened by malinphy about 1 year ago - 3 comments

#156 - Fine-tuned Cross-encoder scores are lower than ms-marco zero-shot scores ?

Issue - State: open - Opened by cramraj8 about 1 year ago - 1 comment

#155 - Support Multi-node evaluation

Pull Request - State: open - Opened by NouamaneTazi over 1 year ago

#154 - Fix `DenseRetrievalExactSearch` evaluation

Pull Request - State: open - Opened by NouamaneTazi over 1 year ago

#153 - Pull latest main branch into development

Pull Request - State: closed - Opened by thakur-nandan over 1 year ago

#152 - Typing for retriever is too strict: implement abstract retriever base class

Issue - State: open - Opened by jon-chuang over 1 year ago

#151 - org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024

Issue - State: open - Opened by 0ddAstronaut over 1 year ago

#150 - Missing test.tsv in thakur/BEIR/datasets/nq-train.zip

Issue - State: closed - Opened by zhiyuanpeng over 1 year ago - 1 comment

#149 - Redirect Anserini BM25 baselines in BEIR to Pyserini

Issue - State: open - Opened by thakur-nandan over 1 year ago

#148 - Merging the development branch with the latest main branch

Pull Request - State: closed - Opened by thakur-nandan over 1 year ago - 2 comments

#147 - Reproduce Signal-1M

Issue - State: closed - Opened by Freddavide over 1 year ago - 2 comments

#146 - evaluate_anserini_bm25.py retrieves 1000 documents for each query no matter which k I set in payload

Issue - State: open - Opened by zhiyuanpeng over 1 year ago - 2 comments

#145 - Don't rebuild bm25 pyserini index every time when running the script

Issue - State: closed - Opened by zhiyuanpeng over 1 year ago

#144 - adding a simple implementation of ColBERT

Issue - State: open - Opened by jjmachan over 1 year ago - 5 comments

#143 - Can't reproduce BM25 baselines

Issue - State: closed - Opened by zhiyuanpeng over 1 year ago - 7 comments

#142 - Update README.md

Pull Request - State: closed - Opened by eltociear over 1 year ago

#141 - Training script/configuration of T5 generator for GenQ

Issue - State: open - Opened by jihyukkim-nlp over 1 year ago - 2 comments

#140 - AssertionError: Elastic-Search Window too large, Max-Size = 10000

Issue - State: open - Opened by zuliani99 over 1 year ago

#139 - [Question] Using T5 as a retriveal model in BeIR

Issue - State: open - Opened by rahmanidashti over 1 year ago - 2 comments

#138 - No module named 'tensorflow_text'

Issue - State: open - Opened by sadanyh over 1 year ago - 3 comments

#137 - Loosen Elasticsearch requirement

Issue - State: open - Opened by stephantul over 1 year ago - 1 comment

#136 - Development

Pull Request - State: closed - Opened by ashokrajab over 1 year ago

#135 - Allow option to disable progress bar when using `GenericDataLoader`

Issue - State: open - Opened by xhluca over 1 year ago - 2 comments

#134 - evaluate_sbert_multi_gpu - metrics.compute() unable to read cache file

Issue - State: open - Opened by ashokrajab over 1 year ago - 1 comment

#133 - Fix BioASQ Avg Q/D according to Paper

Pull Request - State: closed - Opened by timsteuer over 1 year ago

#132 - Can't download data

Issue - State: closed - Opened by TahaBinhuraib over 1 year ago - 2 comments

#131 - Can't explain low results for some models

Issue - State: open - Opened by NebelAI over 1 year ago - 1 comment

#130 - Adding the ANTIQUE dataset to BEIR

Issue - State: open - Opened by heliah over 1 year ago - 4 comments

#129 - Reproducability for all-minilm-l6-v2

Issue - State: open - Opened by Manikseam almost 2 years ago - 1 comment

#128 - Dataset Characteristics Report on `data_loader` Function

Issue - State: open - Opened by rahmanidashti almost 2 years ago

#127 - Allow "show_progress_bar" and "convert_to_tensor" in `DenseRetrievalExactSearch`

Pull Request - State: closed - Opened by xhluca almost 2 years ago - 1 comment

#126 - MonoT5 .predict gives NaN

Issue - State: open - Opened by maxmatical almost 2 years ago - 2 comments

#125 - Use invoked logger and not logging library

Pull Request - State: closed - Opened by ugm2 almost 2 years ago - 1 comment

#124 - Proposal: Describe metrics in Wiki with more details

Issue - State: open - Opened by FabianHertwig almost 2 years ago - 2 comments

#123 - supporting chinese dataset retrieval?

Issue - State: open - Opened by yajingsunno almost 2 years ago - 3 comments

#122 - Training script/configuration for DocT5query

Issue - State: open - Opened by jihyukkim-nlp about 2 years ago

#121 - Quora titles are empty on Huggingface

Issue - State: closed - Opened by jxmorris12 about 2 years ago - 2 comments

#120 - Add file for using custom sbert model with BEIR

Pull Request - State: open - Opened by Agrover112 about 2 years ago

#119 - Add example notebook for benchmarking BEIR using an Sentence Transformer(SBERT) Custom Model

Issue - State: open - Opened by Agrover112 about 2 years ago - 1 comment

#118 - fix: typing issue raising warnings in mypy and pylance

Pull Request - State: open - Opened by qherreros about 2 years ago - 3 comments

#117 - Fix bug of retrieving more docs than specified: Use heapq to maintain top-K

Pull Request - State: closed - Opened by kwang2049 about 2 years ago - 7 comments

#116 - Support for revisions in HFDataLoader

Issue - State: open - Opened by Muennighoff about 2 years ago

#115 - Check tf dependencies completely

Pull Request - State: closed - Opened by kwang2049 about 2 years ago

#114 - HFDataLoader is incompatible with CQADupstack

Issue - State: open - Opened by Muennighoff about 2 years ago

#113 - ModuleNotFoundError

Pull Request - State: closed - Opened by haha978 about 2 years ago

#112 - New Dataset inclusion to BEIR benchmarks

Issue - State: open - Opened by naveenjafer about 2 years ago

#111 - the number of queries is not as reported for some datasets?

Issue - State: closed - Opened by lmh0921 about 2 years ago - 2 comments

#110 - Different version of marco training set

Issue - State: closed - Opened by jzhoubu about 2 years ago - 1 comment

#109 - DenseRetrievalExactSearch: Out of memory

Issue - State: open - Opened by memray about 2 years ago - 1 comment

#108 - NQ training qrels missing

Issue - State: closed - Opened by cadurosar about 2 years ago - 2 comments

#107 - fix cos_scores_top_k_idx when using DRPES

Pull Request - State: closed - Opened by NouamaneTazi over 2 years ago

#106 - Is re_ranking implementation wrong?

Issue - State: open - Opened by wasiahmad over 2 years ago - 5 comments

#105 - Custom metrics handle document score ties differently from pytrec_eval

Issue - State: open - Opened by obelhumeur over 2 years ago

#104 - Different results using DenseRetrievalExactSearch & DenseRetrievalParallelExactSearch

Issue - State: open - Opened by Muennighoff over 2 years ago - 10 comments

#103 - Does BEIR support tuning config k and b of BM25 ?

Issue - State: open - Opened by Yuan0320 over 2 years ago - 2 comments

#102 - custom metric implementation depend on length of qrels

Issue - State: closed - Opened by iknoorjobs over 2 years ago - 2 comments

#101 - Inconsistency in Arguana

Issue - State: open - Opened by claeyzre over 2 years ago

#100 - DPR [KILT] checkpoint release plan and which data is used?

Issue - State: open - Opened by hanseokOh over 2 years ago

#99 - Re-using same GenericDataLoader for different splits raises KeyError

Issue - State: open - Opened by j-rossi-nl over 2 years ago

#98 - Merge all latest development changes to master

Pull Request - State: closed - Opened by thakur-nandan over 2 years ago

#97 - Parallel computation fixes

Pull Request - State: closed - Opened by NouamaneTazi over 2 years ago

#96 - Fix type annotation

Pull Request - State: closed - Opened by jordane95 over 2 years ago - 1 comment

#95 - the number of queries in MSMARCO

Issue - State: closed - Opened by mjeensung over 2 years ago - 2 comments

#94 - Thoughts on reorganizing BEIR main landing page

Issue - State: closed - Opened by lintool over 2 years ago - 2 comments

#93 - `java.lang.OutOfMemoryError: Java heap space` error when running docker

Issue - State: closed - Opened by ichbinhandsome over 2 years ago - 1 comment

#92 - Add Manticore Search to the BEIR benchmarks

Pull Request - State: open - Opened by Nick-S-2018 over 2 years ago

GitHub / beir-cellar/beir issues and pull requests