bigscience-workshop/evaluation issues and pull requests

#82 - build: sync poetry, setup.py, and requirements with safer deps

Pull Request - State: open - Opened by tianjianjiang over 2 years ago

#82 - build: sync poetry, setup.py, and requirements with safer deps

Pull Request - State: open - Opened by tianjianjiang over 2 years ago

#81 - adding arabic sentences for the shade dataset

Pull Request - State: open - Opened by maraimm over 2 years ago

#81 - adding arabic sentences for the shade dataset

Pull Request - State: open - Opened by maraimm over 2 years ago

#80 - adding shades_nationality_hi.csv

Pull Request - State: open - Opened by shanyas10 over 2 years ago

#80 - adding shades_nationality_hi.csv

Pull Request - State: open - Opened by shanyas10 over 2 years ago

#79 - Robustly bump dependency versions

Pull Request - State: closed - Opened by jaketae over 2 years ago - 2 comments

#79 - Robustly bump dependency versions

Pull Request - State: closed - Opened by jaketae over 2 years ago - 2 comments

#78 - bias-shades dataset

Pull Request - State: closed - Opened by manandey over 2 years ago

#78 - bias-shades dataset

Pull Request - State: closed - Opened by manandey over 2 years ago

#77 - Restructuring french dictionary and adding more.

Pull Request - State: closed - Opened by mmitchellai over 2 years ago

#77 - Restructuring french dictionary and adding more.

Pull Request - State: closed - Opened by mmitchellai over 2 years ago

#76 - Adding WebNLG dataset (GEM benchmark)

Pull Request - State: closed - Opened by jordiclive over 2 years ago - 3 comments

#76 - Adding WebNLG dataset (GEM benchmark)

Pull Request - State: closed - Opened by jordiclive over 2 years ago - 3 comments

#75 - Adding WebNLG dataset (GEM benchmark)

Pull Request - State: closed - Opened by jordiclive over 2 years ago

#75 - Adding WebNLG dataset (GEM benchmark)

Pull Request - State: closed - Opened by jordiclive over 2 years ago

#74 - fix: update the version of transformers to support in evaluating gpt-neo models

Pull Request - State: open - Opened by YU-Anthony over 2 years ago - 1 comment

#74 - fix: update the version of transformers to support in evaluating gpt-neo models

Pull Request - State: open - Opened by YU-Anthony over 2 years ago - 1 comment

#73 - Add MKQA to Full Benchmark

Issue - State: open - Opened by shayne-longpre almost 3 years ago

#73 - Add MKQA to Full Benchmark

Issue - State: open - Opened by shayne-longpre almost 3 years ago

#72 - Add BLiMP task

Pull Request - State: open - Opened by jumelet almost 3 years ago

#72 - Add BLiMP task

Pull Request - State: open - Opened by jumelet almost 3 years ago

#71 - Add ANLI dataset

Pull Request - State: open - Opened by omerant almost 3 years ago

#71 - Add ANLI dataset

Pull Request - State: open - Opened by omerant almost 3 years ago

#70 - Add CrowS-Pairs task

Pull Request - State: open - Opened by oskarvanderwal about 3 years ago

#70 - Add CrowS-Pairs task

Pull Request - State: open - Opened by oskarvanderwal about 3 years ago

#69 - Add HANS dataset

Pull Request - State: open - Opened by aakanksha19 about 3 years ago

#69 - Add HANS dataset

Pull Request - State: open - Opened by aakanksha19 about 3 years ago

#68 - feat: Add LAMA TREX task

Pull Request - State: closed - Opened by JanKalo about 3 years ago

#68 - feat: Add LAMA TREX task

Pull Request - State: closed - Opened by JanKalo about 3 years ago

#67 - add jigsaw_toxicity_pred data

Pull Request - State: closed - Opened by trishalaneeraj about 3 years ago - 2 comments
Labels: social_impact

#67 - add jigsaw_toxicity_pred data

Pull Request - State: closed - Opened by trishalaneeraj about 3 years ago - 2 comments
Labels: social_impact

#66 - data path for custom datasets

Pull Request - State: closed - Opened by debajyotidatta about 3 years ago

#66 - data path for custom datasets

Pull Request - State: closed - Opened by debajyotidatta about 3 years ago

#65 - Add XQuAD and PIAF datasets

Pull Request - State: closed - Opened by ryanzhumich about 3 years ago

#65 - Add XQuAD and PIAF datasets

Pull Request - State: closed - Opened by ryanzhumich about 3 years ago

#64 - Refactor task template to merge multilingual.json and english.json

Issue - State: open - Opened by marinecarpuat about 3 years ago

#64 - Refactor task template to merge multilingual.json and english.json

Issue - State: open - Opened by marinecarpuat about 3 years ago

#63 - README cleanup + CONTRIBUTING.md

Pull Request - State: closed - Opened by wilsonyhlee about 3 years ago - 3 comments

#63 - README cleanup + CONTRIBUTING.md

Pull Request - State: closed - Opened by wilsonyhlee about 3 years ago - 3 comments

#62 - feat: use promptsource templates

Pull Request - State: open - Opened by tianjianjiang about 3 years ago

#62 - feat: use promptsource templates

Pull Request - State: open - Opened by tianjianjiang about 3 years ago

#61 - Wrap evaluation benchmark using HF-trainer

Issue - State: open - Opened by sbmaruf about 3 years ago - 2 comments

#61 - Wrap evaluation benchmark using HF-trainer

Issue - State: open - Opened by sbmaruf about 3 years ago - 2 comments

#60 - Add basic template

Pull Request - State: closed - Opened by jaketae about 3 years ago - 1 comment
Labels: enhancement

#60 - Add basic template

Pull Request - State: closed - Opened by jaketae about 3 years ago - 1 comment
Labels: enhancement

#59 - Add PIQA dataset

Pull Request - State: closed - Opened by tttyuntian about 3 years ago
Labels: simple_benchmark

#59 - Add PIQA dataset

Pull Request - State: closed - Opened by tttyuntian about 3 years ago
Labels: simple_benchmark

#58 - Add WMT dataset

Pull Request - State: closed - Opened by jaketae about 3 years ago - 5 comments
Labels: simple_benchmark

#58 - Add WMT dataset

Pull Request - State: closed - Opened by jaketae about 3 years ago - 5 comments
Labels: simple_benchmark

#57 - Setup testing

Issue - State: open - Opened by jaketae about 3 years ago - 2 comments
Labels: engineering

#57 - Setup testing

Issue - State: open - Opened by jaketae about 3 years ago - 2 comments
Labels: engineering

#56 - build: apply a typical project boilerplate of BigScience repo

Pull Request - State: closed - Opened by tianjianjiang about 3 years ago - 1 comment

#56 - build: apply a typical project boilerplate of BigScience repo

Pull Request - State: closed - Opened by tianjianjiang about 3 years ago - 1 comment

#55 - Modify `AutoTask` + adapt `TyDiQA`

Pull Request - State: closed - Opened by jaketae about 3 years ago - 1 comment
Labels: engineering

#55 - Modify `AutoTask` + adapt `TyDiQA`

Pull Request - State: closed - Opened by jaketae about 3 years ago - 1 comment
Labels: engineering

#54 - Start overleaf for benchmark tech report

Issue - State: open - Opened by epavlick about 3 years ago - 1 comment
Labels: documentation

#54 - Start overleaf for benchmark tech report

Issue - State: open - Opened by epavlick about 3 years ago - 1 comment
Labels: documentation

#53 - Refactor evaluation pipeline

Pull Request - State: closed - Opened by jaketae over 3 years ago - 4 comments

#53 - Refactor evaluation pipeline

Pull Request - State: closed - Opened by jaketae over 3 years ago - 4 comments

#52 - Refactor overall directory structure

Pull Request - State: closed - Opened by jaketae over 3 years ago - 6 comments
Labels: engineering

#52 - Refactor overall directory structure

Pull Request - State: closed - Opened by jaketae over 3 years ago - 6 comments
Labels: engineering

#51 - Add LAMBADA evaluation script

Pull Request - State: closed - Opened by jaketae over 3 years ago
Labels: simple_benchmark

#51 - Add LAMBADA evaluation script

Pull Request - State: closed - Opened by jaketae over 3 years ago
Labels: simple_benchmark

#50 - translate validation prompts into all training languages

Issue - State: open - Opened by epavlick over 3 years ago - 9 comments
Labels: simple_benchmark, multilingual

#50 - translate validation prompts into all training languages

Issue - State: open - Opened by epavlick over 3 years ago - 9 comments
Labels: simple_benchmark, multilingual

#49 - benchmark mt5 on tydiqa prompting setup

Issue - State: open - Opened by epavlick over 3 years ago - 1 comment
Labels: simple_benchmark

#49 - benchmark mt5 on tydiqa prompting setup

Issue - State: open - Opened by epavlick over 3 years ago - 1 comment
Labels: simple_benchmark

#48 - Convert validation code to work with Megatron as well as huggingface

Issue - State: open - Opened by epavlick over 3 years ago
Labels: engineering

#48 - Convert validation code to work with Megatron as well as huggingface

Issue - State: open - Opened by epavlick over 3 years ago
Labels: engineering

#47 - Add lambada to validation set

Issue - State: closed - Opened by epavlick over 3 years ago
Labels: simple_benchmark

#47 - Add lambada to validation set

Issue - State: closed - Opened by epavlick over 3 years ago
Labels: simple_benchmark

#46 - Add PIQA to validation set

Issue - State: closed - Opened by epavlick over 3 years ago
Labels: simple_benchmark

#46 - Add PIQA to validation set

Issue - State: closed - Opened by epavlick over 3 years ago
Labels: simple_benchmark

#45 - Adding datasets in full benchmark to the README

Pull Request - State: closed - Opened by wilsonyhlee over 3 years ago

#45 - Adding datasets in full benchmark to the README

Pull Request - State: closed - Opened by wilsonyhlee over 3 years ago

#44 - Add TyDi QA to Simple Benchmark

Pull Request - State: closed - Opened by wilsonyhlee over 3 years ago
Labels: simple_benchmark

#44 - Add TyDi QA to Simple Benchmark

Pull Request - State: closed - Opened by wilsonyhlee over 3 years ago
Labels: simple_benchmark

#43 - Set code conventions

Issue - State: closed - Opened by jaketae over 3 years ago - 3 comments
Labels: engineering

#43 - Set code conventions

Issue - State: closed - Opened by jaketae over 3 years ago - 3 comments
Labels: engineering

#42 - Add WMT evaluation script

Pull Request - State: closed - Opened by jaketae over 3 years ago - 1 comment
Labels: simple_benchmark

#42 - Add WMT evaluation script

Pull Request - State: closed - Opened by jaketae over 3 years ago - 1 comment
Labels: simple_benchmark

#41 - Add XSum to Full Benchmark

Issue - State: closed - Opened by aobaruwa over 3 years ago - 5 comments

#41 - Add XSum to Full Benchmark

Issue - State: closed - Opened by aobaruwa over 3 years ago - 5 comments

#40 - Create toy tasks/dummy code for fine-tuning evals

Issue - State: open - Opened by epavlick over 3 years ago - 1 comment
Labels: engineering

#40 - Create toy tasks/dummy code for fine-tuning evals

Issue - State: open - Opened by epavlick over 3 years ago - 1 comment
Labels: engineering

#39 - Create toy tasks/dummy code for prompt-based evals

Issue - State: closed - Opened by epavlick over 3 years ago - 1 comment
Labels: engineering

#39 - Create toy tasks/dummy code for prompt-based evals

Issue - State: closed - Opened by epavlick over 3 years ago - 1 comment
Labels: engineering

#38 - Create Targeted Minimal Pair "Stress-Tests" for Sensitivity to Social Groups

Issue - State: open - Opened by epavlick over 3 years ago - 2 comments
Labels: social_impact

#38 - Create Targeted Minimal Pair "Stress-Tests" for Sensitivity to Social Groups

Issue - State: open - Opened by epavlick over 3 years ago - 2 comments
Labels: social_impact

#37 - Add CrowS-Pairs to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 5 comments
Labels: social_impact, In-Progress

#37 - Add CrowS-Pairs to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 5 comments
Labels: social_impact, In-Progress

#36 - Add Jigsaw Toxicity Classification to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 4 comments
Labels: social_impact

#36 - Add Jigsaw Toxicity Classification to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 4 comments
Labels: social_impact

#35 - Add WinoMT to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 2 comments
Labels: social_impact

#35 - Add WinoMT to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 2 comments
Labels: social_impact

#34 - Add HANS to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 3 comments
Labels: few_shot

#34 - Add HANS to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 3 comments
Labels: few_shot

#33 - Add MNLI to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 3 comments
Labels: few_shot

#33 - Add MNLI to Full Benchmark

Issue - State: open - Opened by epavlick over 3 years ago - 3 comments
Labels: few_shot

GitHub / bigscience-workshop/evaluation issues and pull requests