AnswerDotAI/bert24 issues and pull requests

#40 - Add flag for applying attn_mask to SDPA

Pull Request - State: open - Opened by warner-benjamin 4 months ago

#39 - Adds code to generate data mixtures with specified proportions.

Pull Request - State: open - Opened by griff4692 4 months ago

#38 - `prefetch_factor` in `convert_dataset.py` should be `None` when not using `num_workers` is not `> 0`

Issue - State: open - Opened by ReinforcedKnowledge 4 months ago

#37 - Migrate ColBERT to conda environment to leverage faiss-gpu

Issue - State: open - Opened by bclavie 4 months ago - 1 comment
Labels: Evals

#36 - Add RoPE with FlexBert blocks

Pull Request - State: closed - Opened by staghado 4 months ago - 2 comments

#35 - Attention fixes

Pull Request - State: closed - Opened by warner-benjamin 4 months ago - 1 comment

#34 - WIP: Superglue 🚧

Pull Request - State: open - Opened by iacolippo 4 months ago

#33 - Parallel attention with flexbert modules

Pull Request - State: closed - Opened by NohTow 4 months ago - 2 comments

#32 - Add FlexBERT, a modular and hackable BERT implementation

Pull Request - State: closed - Opened by warner-benjamin 4 months ago - 4 comments

#31 - Add warmup stable decay lr schedule

Pull Request - State: closed - Opened by ohallstrom 5 months ago - 1 comment

#30 - Code for parralel attention

Pull Request - State: closed - Opened by NohTow 5 months ago - 4 comments

#29 - feat: ColBERT eval

Pull Request - State: closed - Opened by bclavie 5 months ago - 1 comment

#28 - ColBERT eval

Pull Request - State: closed - Opened by bclavie 5 months ago

#27 - add Dolma data sampling code

Pull Request - State: closed - Opened by orionw 5 months ago - 1 comment

#26 - Add DeBERTa baseline

Pull Request - State: closed - Opened by jackcook 5 months ago - 2 comments

#25 - Arch: Implement options to switch between Pre-norm/Post-norm

Issue - State: closed - Opened by bclavie 5 months ago - 2 comments
Labels: Model Arch

#24 - Initial modularization of BERT layers

Pull Request - State: closed - Opened by warner-benjamin 5 months ago

#23 - Modularise BERT layers

Issue - State: closed - Opened by bclavie 5 months ago

#22 - Data Quality Evaluation Suite

Issue - State: open - Opened by rbiswasfc 5 months ago - 3 comments
Labels: Evals

#21 - chore: add optional ablation config object to disable Noam arch changes

Pull Request - State: closed - Opened by bclavie 5 months ago
Labels: Model Arch

#20 - Arch: Add flags to turn model arch changes off to facilitate ablations

Issue - State: closed - Opened by bclavie 5 months ago
Labels: Model Arch

#19 - Broad Training: Design training loop

Issue - State: open - Opened by bclavie 5 months ago - 1 comment
Labels: Training

#18 - Evals: ColBERT(v1) on select BEIR datasets

Issue - State: closed - Opened by bclavie 5 months ago
Labels: Evals

#17 - Arch: Figure out the most inference-efficient model architecture for the XL model size.

Issue - State: open - Opened by bclavie 5 months ago
Labels: Model Arch

#16 - Arch: Figure out the most inference-efficient model architecture for the Large model size.

Issue - State: open - Opened by bclavie 5 months ago
Labels: Model Arch

#15 - Data: Dolma 1.7 to MDS streaming format

Issue - State: open - Opened by bclavie 5 months ago - 3 comments
Labels: Data

#14 - Data: Prepare a 20B tokens subsample of Dolma 1.7

Issue - State: open - Opened by bclavie 5 months ago - 10 comments
Labels: Data

#13 - Add RMSNorm and Swish

Pull Request - State: closed - Opened by bclavie 5 months ago - 2 comments

#12 - Baseline: EncT5

Issue - State: open - Opened by bclavie 5 months ago - 3 comments
Labels: Baselines

#11 - Baseline: DeBERTav3

Issue - State: closed - Opened by bclavie 5 months ago - 2 comments
Labels: DeBERTa, Baselines

#10 - DeBERTa: Add RTD training objective

Issue - State: open - Opened by bclavie 5 months ago - 4 comments
Labels: DeBERTa

#9 - EVALS: Add few-shot evals (SetFIT/FastFIT?)

Issue - State: closed - Opened by bclavie 5 months ago - 1 comment
Labels: Evals

#8 - EVALS: Add SuperGLUE

Issue - State: open - Opened by bclavie 5 months ago
Labels: Evals

#7 - Flash Attention 2 full support

Issue - State: closed - Opened by bclavie 5 months ago
Labels: Model Arch

#6 - Arch: Figure out the most inference-efficient model architecture for the Base model size.

Issue - State: open - Opened by bclavie 5 months ago
Labels: Model Arch

#5 - Implement SwiGLU in lieu of GEGLU

Issue - State: closed - Opened by bclavie 5 months ago
Labels: Model Arch

#4 - RMSNorm instead of Low-Precision LayerNorm

Issue - State: closed - Opened by bclavie 5 months ago
Labels: Model Arch

#3 - Arch: RoPE instead of ALiBi

Issue - State: closed - Opened by bclavie 5 months ago - 2 comments
Labels: Model Arch

#2 - Arch: Parallel Attention

Issue - State: closed - Opened by bclavie 5 months ago
Labels: Model Arch

#1 - Support current libraries and add Flash Attention varlen support

Pull Request - State: closed - Opened by warner-benjamin 5 months ago - 1 comment

GitHub / AnswerDotAI/bert24 issues and pull requests