drisspg/transformer_nuggets issues and pull requests

#39 - Update benchmarks

Pull Request - State: open - Opened by drisspg 25 days ago

#38 - small tweaks

Pull Request - State: open - Opened by drisspg about 2 months ago

#37 - Fixes device-tma kernel

Pull Request - State: closed - Opened by drisspg about 2 months ago

#36 - Adding comparison for different fp8 matmuls

Pull Request - State: closed - Opened by drisspg 2 months ago

#35 - Add some utils for working with flex

Pull Request - State: closed - Opened by drisspg 5 months ago

#34 - add doctsring to profiler

Pull Request - State: closed - Opened by drisspg 6 months ago - 2 comments

#33 - import nanif to init

Pull Request - State: closed - Opened by drisspg 6 months ago - 2 comments

#32 - Updates to ruff

Pull Request - State: closed - Opened by drisspg 7 months ago

#31 - add an option to quantize in chunks

Pull Request - State: closed - Opened by drisspg 7 months ago - 3 comments

#30 - [WIP] enable FSDP x GaLore

Pull Request - State: open - Opened by janeyx99 7 months ago

#29 - Allow for score mod and change of base perf trick

Pull Request - State: closed - Opened by drisspg 8 months ago

#28 - Dynamic scaling triton kernel

Pull Request - State: closed - Opened by drisspg 8 months ago

#27 - Different but equally good updates to flash

Pull Request - State: closed - Opened by drisspg 8 months ago

#26 - Remove dtype restriction and test

Pull Request - State: closed - Opened by drisspg 8 months ago

#25 - Add ShapeLog mode to utilities

Pull Request - State: closed - Opened by drisspg 9 months ago

#24 - Transpose throws an annoying wrench in the mix

Pull Request - State: closed - Opened by drisspg 9 months ago - 2 comments

#23 - fix qlora mlp bug and add script for getting memory traces

Pull Request - State: closed - Opened by drisspg 9 months ago

#22 - Add op table for torch dispatch

Pull Request - State: closed - Opened by drisspg 9 months ago - 1 comment

#21 - [WIP] full finetune / qlora + ac/offload/optm in bwd

Pull Request - State: closed - Opened by weifengpy 9 months ago - 2 comments

#20 - Fix typing and add assert in nf4_tensor

Pull Request - State: closed - Opened by rohan-varma 9 months ago

#19 - Add un-optimized dequant kernel

Pull Request - State: closed - Opened by drisspg 10 months ago

#18 - Make Nf4 a NF4 Tensor subclass

Pull Request - State: closed - Opened by drisspg 10 months ago

#17 - fix_tests

Pull Request - State: closed - Opened by drisspg 10 months ago

#16 - alll the flake8s

Pull Request - State: closed - Opened by drisspg 10 months ago

#15 - enable per-parameter-sharding FSDP + qlora

Pull Request - State: closed - Opened by weifengpy 10 months ago

#14 - added qlora + fsdp

Pull Request - State: closed - Opened by weifengpy 10 months ago

#13 - enable qlora finetuning on single GPU

Pull Request - State: closed - Opened by weifengpy 10 months ago - 1 comment

#12 - add_nan_inf_detect_mode

Pull Request - State: closed - Opened by drisspg 11 months ago

#11 - Pre commit

Pull Request - State: closed - Opened by drisspg 12 months ago

#10 - Add Llama Training scripts

Pull Request - State: closed - Opened by drisspg 12 months ago

#9 - Add benchmark fp8 script

Pull Request - State: closed - Opened by drisspg 12 months ago

#8 - use ufmt on prs

Pull Request - State: closed - Opened by drisspg 12 months ago

#7 - Simple Fp8 delayed scaling kernel

Pull Request - State: closed - Opened by drisspg about 1 year ago

#6 - updated

Pull Request - State: closed - Opened by drisspg about 1 year ago

#5 - Sdpa api

Pull Request - State: closed - Opened by drisspg about 1 year ago - 1 comment

#4 - Update torch.cuda.memory API calls for memory profiling

Pull Request - State: closed - Opened by janeyx99 about 1 year ago

#3 - Block mask

Pull Request - State: closed - Opened by drisspg over 1 year ago

#2 - Update Qlora with end to end llamv2 training

Issue - State: closed - Opened by drisspg over 1 year ago

#1 - Flash Attention V2 w/ arbitrary attention bias

Pull Request - State: closed - Opened by drisspg over 1 year ago

GitHub / drisspg/transformer_nuggets issues and pull requests