Ecosyste.ms: Issues
An open API service for providing issue and pull request metadata for open source projects.
GitHub / huggingface/evaluate issues and pull requests
#539 - [metric] macro and weighted average metrics for seqeval
Pull Request -
State: open - Opened by imvladikon 8 months ago
#538 - Remove an unused dependency
Pull Request -
State: closed - Opened by yasufumy 8 months ago
#537 - Is TextGenerationEvaluator incomplete?
Issue -
State: open - Opened by inf3rnus 9 months ago
- 5 comments
#536 - Fix toxicity and regard measurements to default to GPU inference
Pull Request -
State: open - Opened by xangelix 9 months ago
#535 - Fixing `scikit-learn` import message to match PyPI recommendation
Pull Request -
State: open - Opened by jamesbraza 9 months ago
#534 - How should i cite this repository?
Issue -
State: open - Opened by perlitz 9 months ago
- 2 comments
#533 - support multilabel confusion matrix
Pull Request -
State: open - Opened by 0ssamaak0 9 months ago
- 7 comments
#532 - `perplexity` yields `IndexError: index out of range in self`
Issue -
State: open - Opened by pminervini 9 months ago
- 1 comment
#531 - Update requirements.txt
Pull Request -
State: open - Opened by pminervini 9 months ago
#530 - Fix typos ; Update gradio.py
Pull Request -
State: open - Opened by furkanakkurt1335 9 months ago
#529 - Bug in BLEU calculation - BLEU of identical strings should be 1.0, not 0.0
Issue -
State: open - Opened by MislavJuric 9 months ago
- 2 comments
#528 - Add confusion matrix
Pull Request -
State: closed - Opened by osanseviero 10 months ago
- 5 comments
#527 - Cannot use evaluate.TextClassificationEvaluator for multi-label classification
Issue -
State: closed - Opened by lorenzobalzani 10 months ago
- 2 comments
#526 - Using huggingface evaluate for distributed evaluation
Issue -
State: open - Opened by plamb-viso 10 months ago
#525 - Getting error <lambda>() takes 0 positional arguments but 1 was given when trying to import evaluator
Issue -
State: open - Opened by DanielMoreiraPinto 10 months ago
#524 - Added index.html
Pull Request -
State: closed - Opened by tybrs 10 months ago
#523 - os.unlink(entry.name, dir_fd=topfd) TypeError: 'NoneType' object is not callable
Issue -
State: open - Opened by starlitsky2010 10 months ago
#522 - DefaultCPUAllocator: can't allocate memory during perplexity.compute for Llama 2
Issue -
State: open - Opened by RinaAkay 10 months ago
#521 - Added the ability to return the dictionary to the WER metric.
Pull Request -
State: open - Opened by blademoon 10 months ago
#520 - Add custom device selection such as 'cuda:1' for perplexity
Pull Request -
State: open - Opened by thatmee 10 months ago
#519 - `evaluate-cli create` fails with `subprocess.CalledProcessError`
Issue -
State: open - Opened by Tai-Mai 10 months ago
- 1 comment
#518 - Importing Error for evaluator module
Issue -
State: open - Opened by DreF174 10 months ago
#517 - Cannot list evaluation modules types
Issue -
State: open - Opened by trajepl 10 months ago
#516 - Can't calculate combined metric (WER + CER)
Issue -
State: open - Opened by blademoon 11 months ago
- 2 comments
#515 - Allow BLEURT optimisation parameters
Issue -
State: open - Opened by Limess 11 months ago
#514 - HuggingFace evaluate-cli create command results Error: "datasets[0]" must be a string
Issue -
State: open - Opened by shivanraptor 11 months ago
- 2 comments
#513 - add_batch taking very long time (SegFormer semantic segmentation).
Issue -
State: open - Opened by itish-align 11 months ago
#512 - about evaluate/perplexity
Issue -
State: open - Opened by KangkangStu 11 months ago
- 2 comments
#511 - Loading metrics that are shipped with the evaluate package is slow
Issue -
State: open - Opened by harmbuisman 11 months ago
- 2 comments
#510 - fix: remove useless token
Pull Request -
State: closed - Opened by rtrompier 12 months ago
#509 - [metric] CRPS metric
Pull Request -
State: open - Opened by kashif 12 months ago
#508 - docs: specify micro average in seqeval README
Pull Request -
State: open - Opened by WojciechKusa 12 months ago
#507 - Remove unused dependency
Pull Request -
State: closed - Opened by daskol 12 months ago
- 2 comments
#506 - set dev version
Pull Request -
State: closed - Opened by lhoestq 12 months ago
#505 - Release: 0.4.1
Pull Request -
State: closed - Opened by lhoestq 12 months ago
#504 - Compatibility Issue with datasets library
Issue -
State: open - Opened by adldotori 12 months ago
- 3 comments
#503 - Move away from `Repository` to `HfApi` upload files
Issue -
State: open - Opened by Wauplin 12 months ago
#502 - Perplexity Space Broken
Issue -
State: closed - Opened by alyserecord 12 months ago
#501 - wrong rouge-Lsum results due to lack of parameter control
Issue -
State: open - Opened by dsashulya 12 months ago
#500 - Wrong accuracy on the validation set!!!
Issue -
State: open - Opened by utility-aagrawal 12 months ago
- 1 comment
#499 - removed unnecessary `module_type` attribute
Pull Request -
State: open - Opened by leo-gan 12 months ago
#498 - fix scikit-learn package name suggestion
Pull Request -
State: closed - Opened by bzz about 1 year ago
- 1 comment
#497 - Multi-label with f1
Issue -
State: closed - Opened by jingmingzhuo about 1 year ago
- 2 comments
#496 - when using custom models for perplexity metrix, support trust_remote_code
Pull Request -
State: open - Opened by winglian about 1 year ago
#495 - CER.py error in class CERConfig
Issue -
State: open - Opened by danhertztech about 1 year ago
- 1 comment
#494 - is it possible to evaluate a proprietary model without uploading it to HF
Issue -
State: open - Opened by sermolin about 1 year ago
- 7 comments
#493 - Feature request for metric result keys in `MetricInfo`
Issue -
State: open - Opened by tybrs about 1 year ago
#492 - Matthew's Correlation Coefficient (MCC) always zero
Issue -
State: closed - Opened by mirix about 1 year ago
- 1 comment
#491 - perplexity.compute - Add args to pass model and tokenizer objects instead of `model_id`
Issue -
State: open - Opened by albernar about 1 year ago
- 1 comment
#490 - Fix `datasets` import in Meteor metric
Pull Request -
State: closed - Opened by mariosasko about 1 year ago
#489 - Concurrent Programs Sharing Evaluate Lib Would raise "ValueError: Error in finalize: another evaluation module instance is already using the local cache file."
Issue -
State: open - Opened by 2catycm about 1 year ago
- 6 comments
#488 - Add support for glue metric in `EvaluationSuite`
Pull Request -
State: open - Opened by tybrs about 1 year ago
- 1 comment
#487 - Evaluate a quantize model
Issue -
State: open - Opened by Psancs05 about 1 year ago
- 2 comments
#486 - evaluate.load("wer") is returning Filenotfound error
Issue -
State: open - Opened by valmetisrinivas about 1 year ago
- 3 comments
#485 - How to use `SubTask` with metrics that require valid `config_name`
Issue -
State: open - Opened by tybrs about 1 year ago
#484 - Metric returned instead of the actual COMET scores
Issue -
State: open - Opened by LeonardoEmili about 1 year ago
- 6 comments
#483 - `.compute()` not compatible with `perplexity` for `text-generation`
Issue -
State: open - Opened by hans-ekbrand about 1 year ago
- 1 comment
#482 - Adding tokenizer_id to perplexity evaluation
Pull Request -
State: open - Opened by Albertoimpl about 1 year ago
- 3 comments
#481 - [Metrics] Can't find / acquire lock files in distributed multi-node shared file system
Issue -
State: open - Opened by Natooz about 1 year ago
- 3 comments
#480 - meteor don't works with latest datasets
Issue -
State: closed - Opened by leoleoasd about 1 year ago
- 13 comments
#479 - Fix formatting of model names in radar plot
Pull Request -
State: open - Opened by potamides about 1 year ago
#478 - Trainer Custom_loss - Empty metrics in Evaluate
Issue -
State: closed - Opened by AkimParis about 1 year ago
- 2 comments
#477 - Question: Computing utterance-level and corpus level metrics
Issue -
State: open - Opened by roshansh-cmu about 1 year ago
#476 - Change package name from "evaluate" to something less generic
Issue -
State: open - Opened by geajack about 1 year ago
#475 - FileNotFoundError: Couldn't find a module script at /data1/yangyan/code/mms_finetune/wer/wer.py. Module 'wer' doesn't exist on the Hugging Face Hub either
Issue -
State: open - Opened by yangy1992 about 1 year ago
- 5 comments
#474 - Fix SQuAD v2 metric docs on `references` format
Pull Request -
State: open - Opened by bryant1410 about 1 year ago
#473 - evaluate.load loads indefinitely offline
Issue -
State: open - Opened by Virgile-Richard1 about 1 year ago
- 1 comment
#472 - [BUG] Loading metrics is extremely slow
Issue -
State: open - Opened by NightMachinery about 1 year ago
- 6 comments
#471 - Fix mean computation in Mahalanobis distance
Pull Request -
State: open - Opened by j0ma about 1 year ago
#470 - Mahalanobis distance computes X_minus_mu incorrectly
Issue -
State: open - Opened by j0ma about 1 year ago
#469 - Compute BLEU score of a Pandas DataFrame with valid rows filtered
Issue -
State: open - Opened by shivanraptor over 1 year ago
#468 - [doc build] Use secrets
Pull Request -
State: open - Opened by mishig25 over 1 year ago
#467 - Trigger CI on ci-* branches
Pull Request -
State: closed - Opened by Wauplin over 1 year ago
- 1 comment
#466 - [don't merge] Test hfh v0.15.0.rc0
Pull Request -
State: closed - Opened by Wauplin over 1 year ago
- 2 comments
#465 - Rouge evaluator produced `0` score for the same prediction and reference in Chinese.
Issue -
State: open - Opened by Whisht over 1 year ago
- 1 comment
#464 - does rouge_score = evaluate.load("rouge") support other languages?
Issue -
State: open - Opened by lixnvege over 1 year ago
- 1 comment
#463 - Render Text2TextGenerationEvaluators' docstring examples
Pull Request -
State: closed - Opened by mariosasko over 1 year ago
- 1 comment
#462 - Merged PR #425 doesn't properly address the issue of passing metric specific kwargs to compute or load
Issue -
State: open - Opened by christian-storm over 1 year ago
#461 - How to disable NLTK download information output?
Issue -
State: open - Opened by yiyualt over 1 year ago
- 1 comment
#460 - Evaluate SyntaxError: invalid syntax
Issue -
State: open - Opened by NLBONU1 over 1 year ago
- 1 comment
#459 - feat: custom evaluator and metric name to support llm evaluation #433
Pull Request -
State: open - Opened by fecet over 1 year ago
#458 - Removed unused and restrictive `responses` requirement
Pull Request -
State: closed - Opened by twerkmeister over 1 year ago
- 2 comments
#457 - Quick tour typos
Issue -
State: open - Opened by trajepl over 1 year ago
#456 - Module 'accuracy' doesn't exist on the Hugging Face Hub either.
Issue -
State: open - Opened by AceCHQ over 1 year ago
- 19 comments
#455 - AttributeError: module 'sacrebleu' has no attribute '__version__'
Issue -
State: closed - Opened by dsj96 over 1 year ago
- 4 comments
#454 - Toxicity Measurement Should Allow GPU Usage
Issue -
State: open - Opened by dni138 over 1 year ago
#453 - SQuAD v2 is missing the NoAns metrics
Issue -
State: open - Opened by GitMew over 1 year ago
#452 - unable to import evaluate
Issue -
State: closed - Opened by Ananthzeke over 1 year ago
- 1 comment
#451 - Huggingface bertscore metric GPU leak?
Issue -
State: open - Opened by will-maclean over 1 year ago
#450 - Adding new metrics locally without using the evaluate-cli command
Issue -
State: closed - Opened by sameeravithana over 1 year ago
- 1 comment
#449 - Metrics that use DownloadConfig not working any more
Issue -
State: closed - Opened by BramVanroy over 1 year ago
- 5 comments
#448 - Add the support to pass a pre-loaded model and a pre-loaded tokenizer
Pull Request -
State: open - Opened by JulesGM over 1 year ago
#447 - replace evaluate DownloadConfig with datasets
Pull Request -
State: closed - Opened by lvwerra over 1 year ago
- 2 comments
#446 - Add kendall tau distance (not as a submodule)
Pull Request -
State: open - Opened by unna97 over 1 year ago
- 3 comments
#445 - Understanding metric compute in DDP setup
Issue -
State: closed - Opened by Natooz over 1 year ago
- 1 comment
#444 - combined metrics in evaluators/Suite subtasks
Issue -
State: open - Opened by Vipitis over 1 year ago
#443 - Update comet
Pull Request -
State: closed - Opened by ricardorei over 1 year ago
- 6 comments
#442 - Progress bar when using evaluator
Issue -
State: open - Opened by zxcej over 1 year ago
- 1 comment
#441 - Problem about code_eval
Issue -
State: closed - Opened by rongaoli over 1 year ago
#440 - AttributeError: 'DownloadConfig' object has no attribute 'storage_options'
Issue -
State: closed - Opened by lhoestq over 1 year ago