saattrupdan/ScandEval issues and pull requests

#73 - Add text modelling benchmark task

Issue - State: open - Opened by saattrupdan about 1 year ago
Labels: enhancement

#72 - Add multiple-choice text generation benchmark functionality

Issue - State: open - Opened by saattrupdan about 1 year ago
Labels: enhancement

#71 - Add NLG benchmark support

Pull Request - State: open - Opened by saattrupdan about 1 year ago

#70 - feat: Add 8-bit AdamW optimizer

Pull Request - State: open - Opened by saattrupdan about 1 year ago

#69 - Fix/do not use auto batch size

Pull Request - State: closed - Opened by saattrupdan about 1 year ago

#68 - enable mistral

Pull Request - State: open - Opened by peter-sk about 1 year ago - 2 comments

#67 - Add model: roberta-base-danish

Issue - State: closed - Opened by KennethEnevoldsen about 1 year ago - 2 comments

#66 - Feat/enable multi gpu

Pull Request - State: open - Opened by saattrupdan about 1 year ago - 1 comment

#65 - Not able to run scandeval with openai models

Issue - State: closed - Opened by marcelcastrobr about 1 year ago - 3 comments

#64 - enable Apple silicone benchmarking

Pull Request - State: closed - Opened by peter-sk about 1 year ago - 2 comments

#63 - Bug when evaluating `peter-sk/gpt-neox-da-tiny`

Issue - State: closed - Opened by saattrupdan over 1 year ago - 1 comment
Labels: bug

#62 - Bug in maximum sequence length

Issue - State: closed - Opened by KennethEnevoldsen over 1 year ago - 1 comment

#61 - Fix/GitHub ci torch bug

Pull Request - State: closed - Opened by saattrupdan over 1 year ago

#60 - feat: Allows parallel benchmarks

Pull Request - State: closed - Opened by KennethEnevoldsen over 1 year ago - 3 comments

#59 - Cache interference issues when testing two models at once

Issue - State: closed - Opened by KennethEnevoldsen over 1 year ago - 2 comments
Labels: enhancement

#58 - Add popular sentence-embedding models

Issue - State: closed - Opened by KennethEnevoldsen over 1 year ago - 3 comments

#57 - Invalid model name does not raise an error or a warning

Issue - State: closed - Opened by KennethEnevoldsen over 1 year ago - 4 comments

#56 - add cls_token/cls_token_id

Pull Request - State: open - Opened by peter-sk over 1 year ago

#55 - left-padding-is-broken in tokenizers

Pull Request - State: closed - Opened by peter-sk over 1 year ago - 2 comments

#54 - gpt-neox models cannot be benchmarked

Issue - State: closed - Opened by peter-sk over 1 year ago - 3 comments
Labels: bug

#53 - option to fix embedding / ability to override model config

Pull Request - State: closed - Opened by peter-sk over 1 year ago - 6 comments

#52 - flax-community/dansk-gpt-wiki could not be benchmarked on the truncated version of AngryTweets

Issue - State: closed - Opened by peter-sk over 1 year ago - 4 comments

#51 - `sentence-transformers/distiluse-base-multilingual-cased-v2` fails during evaluation

Issue - State: closed - Opened by saattrupdan almost 2 years ago - 1 comment
Labels: bug

#50 - CANINE models cannot be benchmarked on QA tasks

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: bug

#49 - LUKE models cannot be benchmarked on NER tasks

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: bug

#48 - Feat/add speed test

Pull Request - State: closed - Opened by saattrupdan almost 2 years ago

#47 - Unit tests for the `benchmark_config_factory` module

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: good first issue, tests

#46 - Unit tests for the `question_answering_trainer` module

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: good first issue, tests

#45 - Unit tests for the `model_loading` module

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: good first issue, tests

#44 - Unit tests for the `benchmarker` module

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: good first issue, tests

#43 - Unit tests for the `benchmark_dataset` module

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: good first issue, tests

#42 - Add support for NLG evaluation

Issue - State: open - Opened by saattrupdan almost 2 years ago
Labels: enhancement

#41 - Add support for zero and few-shot evaluation

Issue - State: closed - Opened by saattrupdan almost 2 years ago
Labels: enhancement

#40 - Add support for SLU evaluation

Issue - State: open - Opened by saattrupdan almost 2 years ago - 3 comments
Labels: enhancement

#39 - Add inference speed evaluation

Issue - State: closed - Opened by saattrupdan almost 2 years ago
Labels: enhancement

#38 - Add a scandeval callback to allow for evaluation during training

Issue - State: open - Opened by KennethEnevoldsen almost 2 years ago
Labels: enhancement, good first issue

#37 - Add proper output when using multiple GPUs

Issue - State: open - Opened by saattrupdan about 2 years ago - 2 comments
Labels: bug

#36 - Update README.md

Pull Request - State: closed - Opened by peregilk about 2 years ago

#35 - Language flag does not work as intended

Issue - State: closed - Opened by KennethEnevoldsen over 2 years ago - 4 comments
Labels: enhancement

#34 - When you misspecify the model name, you don't get an error

Issue - State: closed - Opened by KennethEnevoldsen over 2 years ago - 4 comments
Labels: enhancement

#33 - Flag --model_id does not work, but -m does

Issue - State: closed - Opened by KennethEnevoldsen over 2 years ago - 1 comment
Labels: bug

#32 - Add support for seq2seq models

Issue - State: closed - Opened by saattrupdan over 2 years ago - 1 comment
Labels: enhancement

#31 - Add mT5

Issue - State: closed - Opened by KennethEnevoldsen over 2 years ago - 4 comments

#30 - Add mdeberta-v3-base

Issue - State: closed - Opened by KennethEnevoldsen over 2 years ago - 2 comments
Labels: enhancement

#29 - Black format

Pull Request - State: closed - Opened by kasperjunge over 2 years ago - 3 comments

#28 - black formatted

Pull Request - State: closed - Opened by kasperjunge over 2 years ago - 1 comment

#27 - Add support for model revisions

Pull Request - State: closed - Opened by versae almost 3 years ago - 1 comment

#26 - Add support for Flax/Jax models

Pull Request - State: closed - Opened by versae almost 3 years ago

#25 - Evaluating private HuggingFace models

Issue - State: closed - Opened by peregilk almost 3 years ago - 1 comment
Labels: enhancement

#24 - Dependency parsing is not being trained optimally

Issue - State: closed - Opened by saattrupdan over 3 years ago - 3 comments
Labels: enhancement

#23 - Benchmarking is not shown properly in notebooks

Issue - State: closed - Opened by saattrupdan over 3 years ago - 2 comments
Labels: bug