mlfoundations/open_lm issues and pull requests

#50 - Adding unites for grad_accum (with and without fsdp)

Pull Request - State: closed - Opened by jfisher52 about 1 year ago

#50 - Adding unites for grad_accum (with and without fsdp)

Pull Request - State: closed - Opened by jfisher52 about 1 year ago

#49 - Multinode Working Tokenize shuffle

Pull Request - State: closed - Opened by Vaishaal about 1 year ago - 1 comment

#49 - Multinode Working Tokenize shuffle

Pull Request - State: closed - Opened by Vaishaal about 1 year ago - 1 comment

#48 - Use no_sync when doing gradient accumulation

Issue - State: closed - Opened by achalddave about 1 year ago

#48 - Use no_sync when doing gradient accumulation

Issue - State: closed - Opened by achalddave about 1 year ago

#47 - Add accumulation test and minor fix

Pull Request - State: closed - Opened by achalddave about 1 year ago - 1 comment

#47 - Add accumulation test and minor fix

Pull Request - State: closed - Opened by achalddave about 1 year ago - 1 comment

#46 - Log and skip errors in map_dict

Pull Request - State: closed - Opened by achalddave about 1 year ago - 6 comments

#46 - Log and skip errors in map_dict

Pull Request - State: closed - Opened by achalddave about 1 year ago - 6 comments

#45 - Convert LLaMA weights to open_lm compatible weights

Pull Request - State: closed - Opened by yuhui-zh15 about 1 year ago - 1 comment

#45 - Convert LLaMA weights to open_lm compatible weights

Pull Request - State: closed - Opened by yuhui-zh15 about 1 year ago - 1 comment

#44 - Black format openlm

Issue - State: closed - Opened by achalddave about 1 year ago

#44 - Black format openlm

Issue - State: closed - Opened by achalddave about 1 year ago

#43 - Speed up loading remote checkpoints

Issue - State: closed - Opened by achalddave about 1 year ago
Labels: good first issue

#43 - Speed up loading remote checkpoints

Issue - State: closed - Opened by achalddave about 1 year ago
Labels: good first issue

#42 - Fix optim_state_dict_to_load call for latest pytorch versions

Pull Request - State: closed - Opened by achalddave about 1 year ago - 1 comment

#42 - Fix optim_state_dict_to_load call for latest pytorch versions

Pull Request - State: closed - Opened by achalddave about 1 year ago - 1 comment

#41 - Documentation: competing frameworks

Issue - State: open - Opened by borgr about 1 year ago - 3 comments

#41 - Documentation: competing frameworks

Issue - State: open - Opened by borgr about 1 year ago - 3 comments

#40 - Fused RMSNorm

Pull Request - State: open - Opened by sagadre about 1 year ago - 2 comments

#40 - Fused RMSNorm

Pull Request - State: open - Opened by sagadre about 1 year ago - 2 comments

#39 - LLaMA2 weight loading

Pull Request - State: closed - Opened by jmercat about 1 year ago - 2 comments

#39 - LLaMA2 weight loading

Pull Request - State: closed - Opened by jmercat about 1 year ago - 2 comments

#38 - Overloading --model param to support passing a json model config

Pull Request - State: closed - Opened by sagadre about 1 year ago

#38 - Overloading --model param to support passing a json model config

Pull Request - State: closed - Opened by sagadre about 1 year ago

#37 - fp8 training on h100s

Issue - State: open - Opened by mitchellnw about 1 year ago

#37 - fp8 training on h100s

Issue - State: open - Opened by mitchellnw about 1 year ago

#36 - Support LLaMA-2 with Backward Compatibility

Pull Request - State: closed - Opened by yuhui-zh15 about 1 year ago

#36 - Support LLaMA-2 with Backward Compatibility

Pull Request - State: closed - Opened by yuhui-zh15 about 1 year ago

#35 - Weird memory usage for 11m vs 160m: similar batch size fits in memory...

Issue - State: open - Opened by alexjc about 1 year ago - 14 comments

#35 - Weird memory usage for 11m vs 160m: similar batch size fits in memory...

Issue - State: open - Opened by alexjc about 1 year ago - 14 comments

#34 - Add bootstrap CIs to val perplexity calculation

Issue - State: closed - Opened by achalddave about 1 year ago

#34 - Add bootstrap CIs to val perplexity calculation

Issue - State: closed - Opened by achalddave about 1 year ago

#33 - [DON'T MERGE] Support llama-2

Pull Request - State: closed - Opened by yuhui-zh15 about 1 year ago - 1 comment

#33 - [DON'T MERGE] Support llama-2

Pull Request - State: closed - Opened by yuhui-zh15 about 1 year ago - 1 comment

#32 - Remove fused cross-entropy

Pull Request - State: closed - Opened by achalddave about 1 year ago

#32 - Remove fused cross-entropy

Pull Request - State: closed - Opened by achalddave about 1 year ago

#31 - Ablate on initialization

Issue - State: closed - Opened by mitchellnw about 1 year ago - 2 comments

#31 - Ablate on initialization

Issue - State: closed - Opened by mitchellnw about 1 year ago - 2 comments

#30 - LLaMA weight loading

Issue - State: closed - Opened by ludwigschmidt about 1 year ago - 2 comments

#30 - LLaMA weight loading

Issue - State: closed - Opened by ludwigschmidt about 1 year ago - 2 comments

#29 - Training speed ups: +10-15% tokens/second/gpu

Pull Request - State: closed - Opened by achalddave about 1 year ago - 2 comments

#29 - Training speed ups: +10-15% tokens/second/gpu

Pull Request - State: closed - Opened by achalddave about 1 year ago - 2 comments

#28 - Fixed up generate.py, added to readme

Pull Request - State: closed - Opened by revbucket about 1 year ago

#28 - Fixed up generate.py, added to readme

Pull Request - State: closed - Opened by revbucket about 1 year ago

#27 - Consider low precision normalization

Issue - State: closed - Opened by mitchellnw about 1 year ago

#27 - Consider low precision normalization

Issue - State: closed - Opened by mitchellnw about 1 year ago

#26 - Mup

Pull Request - State: closed - Opened by reinhardh about 1 year ago - 1 comment

#26 - Mup

Pull Request - State: closed - Opened by reinhardh about 1 year ago - 1 comment

#25 - mup simple

Pull Request - State: closed - Opened by sagadre about 1 year ago - 1 comment

#25 - mup simple

Pull Request - State: closed - Opened by sagadre about 1 year ago - 1 comment

#24 - Consolidate eval arguments with train arguments

Pull Request - State: closed - Opened by achalddave about 1 year ago - 3 comments

#24 - Consolidate eval arguments with train arguments

Pull Request - State: closed - Opened by achalddave about 1 year ago - 3 comments

#23 - Add support for sparse mixture of experts (MoE)

Issue - State: closed - Opened by sagadre about 1 year ago - 1 comment

#23 - Add support for sparse mixture of experts (MoE)

Issue - State: closed - Opened by sagadre about 1 year ago - 1 comment

#22 - add fused cross entropy

Issue - State: closed - Opened by kernelmachine about 1 year ago

#22 - add fused cross entropy

Issue - State: closed - Opened by kernelmachine about 1 year ago

#21 - Benchmark tok/sec with other libs

Issue - State: open - Opened by kernelmachine about 1 year ago - 4 comments

#21 - Benchmark tok/sec with other libs

Issue - State: open - Opened by kernelmachine about 1 year ago - 4 comments

#20 - Add HF mixin

Pull Request - State: closed - Opened by NielsRogge about 1 year ago - 7 comments

#20 - Add HF mixin

Pull Request - State: closed - Opened by NielsRogge about 1 year ago - 7 comments

#19 - Release the intermediate checkpoints?

Issue - State: open - Opened by NathanGodey about 1 year ago - 2 comments

#19 - Release the intermediate checkpoints?

Issue - State: open - Opened by NathanGodey about 1 year ago - 2 comments

#18 - Add doc+support for evaluating with rotary-old

Pull Request - State: closed - Opened by achalddave about 1 year ago

#18 - Add doc+support for evaluating with rotary-old

Pull Request - State: closed - Opened by achalddave about 1 year ago

#17 - Vq gpt adaptation

Pull Request - State: closed - Opened by iejMac about 1 year ago

#17 - Vq gpt adaptation

Pull Request - State: closed - Opened by iejMac about 1 year ago

#16 - main.py: use data_key in get_wds_dataset

Pull Request - State: closed - Opened by iejMac about 1 year ago

#16 - main.py: use data_key in get_wds_dataset

Pull Request - State: closed - Opened by iejMac about 1 year ago

#15 - datapreprocesss: make_2048.py zero-pad sample ID

Pull Request - State: closed - Opened by iejMac about 1 year ago

#15 - datapreprocesss: make_2048.py zero-pad sample ID

Pull Request - State: closed - Opened by iejMac about 1 year ago

#14 - VQ-GPT

Pull Request - State: closed - Opened by iejMac about 1 year ago - 1 comment

#14 - VQ-GPT

Pull Request - State: closed - Opened by iejMac about 1 year ago - 1 comment

#13 - Replaced pile -> wikipedia in quickstart instructions

Pull Request - State: closed - Opened by revbucket about 1 year ago

#13 - Replaced pile -> wikipedia in quickstart instructions

Pull Request - State: closed - Opened by revbucket about 1 year ago

#12 - Add generation scripts

Pull Request - State: closed - Opened by achalddave about 1 year ago

#12 - Add generation scripts

Pull Request - State: closed - Opened by achalddave about 1 year ago

#11 - Revamp make_2048.py script

Issue - State: closed - Opened by sagadre about 1 year ago - 3 comments

#11 - Revamp make_2048.py script

Issue - State: closed - Opened by sagadre about 1 year ago - 3 comments

#10 - train.py: sample index in chunk when chunk_size >= seq_len

Pull Request - State: closed - Opened by iejMac about 1 year ago

#10 - train.py: sample index in chunk when chunk_size >= seq_len

Pull Request - State: closed - Opened by iejMac about 1 year ago

#9 - Unit test for grad accum

Issue - State: closed - Opened by mitchellnw about 1 year ago - 2 comments

#9 - Unit test for grad accum

Issue - State: closed - Opened by mitchellnw about 1 year ago - 2 comments

#8 - Improved dataloading with multiple sources without resampling.

Pull Request - State: closed - Opened by GeorgiosSmyrnis about 1 year ago - 4 comments

#8 - Improved dataloading with multiple sources without resampling.

Pull Request - State: closed - Opened by GeorgiosSmyrnis about 1 year ago - 4 comments

#7 - Exit on NaN loss

Pull Request - State: closed - Opened by sagadre about 1 year ago - 3 comments

#7 - Exit on NaN loss

Pull Request - State: closed - Opened by sagadre about 1 year ago - 3 comments

#6 - Hybrid sharding change

Pull Request - State: closed - Opened by GeorgiosSmyrnis about 1 year ago

#5 - Positional embedding xformers fix

Pull Request - State: closed - Opened by sagadre about 1 year ago - 2 comments

#4 - Problem in position embedding

Issue - State: closed - Opened by jmercat about 1 year ago - 8 comments

#3 - Add a fused RMSNorm operation

Issue - State: closed - Opened by mitchellnw about 1 year ago - 4 comments

#2 - Update README.md

Pull Request - State: closed - Opened by revbucket about 1 year ago

#1 - pip installable

Pull Request - State: closed - Opened by sagadre about 1 year ago

GitHub / mlfoundations/open_lm issues and pull requests