feifeibear/long-context-attention issues and pull requests

#33 - process group as global vars

Issue - State: closed - Opened by feifeibear 8 months ago

#33 - process group as global vars

Issue - State: closed - Opened by feifeibear 8 months ago

#32 - reorganize the directories

Pull Request - State: closed - Opened by feifeibear 8 months ago

#32 - reorganize the directories

Pull Request - State: closed - Opened by feifeibear 8 months ago

#32 - reorganize the directories

Pull Request - State: closed - Opened by feifeibear 8 months ago

#31 - fix ulysses bugs

Pull Request - State: closed - Opened by feifeibear 8 months ago

#31 - fix ulysses bugs

Pull Request - State: closed - Opened by feifeibear 8 months ago

#31 - fix ulysses bugs

Pull Request - State: closed - Opened by feifeibear 8 months ago

#30 - update benchmark scripts

Pull Request - State: closed - Opened by feifeibear 8 months ago

#30 - update benchmark scripts

Pull Request - State: closed - Opened by feifeibear 8 months ago

#30 - update benchmark scripts

Pull Request - State: closed - Opened by feifeibear 8 months ago

#29 - 请问下，这几类方法可以与Deepspeed Zero一块使用吗？

Issue - State: closed - Opened by YouYouCoding 8 months ago - 6 comments

#29 - 请问下，这几类方法可以与Deepspeed Zero一块使用吗？

Issue - State: closed - Opened by YouYouCoding 8 months ago - 6 comments

#29 - 请问下，这几类方法可以与Deepspeed Zero一块使用吗？

Issue - State: closed - Opened by YouYouCoding 8 months ago - 6 comments

#28 - add use_ulysses_low flag

Pull Request - State: closed - Opened by feifeibear 8 months ago

#28 - add use_ulysses_low flag

Pull Request - State: closed - Opened by feifeibear 8 months ago

#28 - add use_ulysses_low flag

Pull Request - State: closed - Opened by feifeibear 8 months ago

#27 - update readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#27 - update readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#27 - update readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#26 - update readme using more ring-attn-impl for LongContextAttention

Pull Request - State: closed - Opened by feifeibear 8 months ago

#26 - update readme using more ring-attn-impl for LongContextAttention

Pull Request - State: closed - Opened by feifeibear 8 months ago

#26 - update readme using more ring-attn-impl for LongContextAttention

Pull Request - State: closed - Opened by feifeibear 8 months ago

#25 - add ring-attn impl type

Pull Request - State: closed - Opened by feifeibear 8 months ago

#25 - add ring-attn impl type

Pull Request - State: closed - Opened by feifeibear 8 months ago

#25 - add ring-attn impl type

Pull Request - State: closed - Opened by feifeibear 8 months ago

#24 - The impact of head number

Issue - State: closed - Opened by GeneZC 8 months ago - 2 comments

#24 - The impact of head number

Issue - State: closed - Opened by GeneZC 8 months ago - 2 comments

#24 - The impact of head number

Issue - State: closed - Opened by GeneZC 8 months ago - 2 comments

#23 - polish readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#23 - polish readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#23 - polish readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#22 - polish readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#22 - polish readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#22 - polish readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#21 - update readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#21 - update readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#20 - add media

Pull Request - State: closed - Opened by feifeibear 8 months ago

#20 - add media

Pull Request - State: closed - Opened by feifeibear 8 months ago

#19 - polish benchmark flags

Pull Request - State: closed - Opened by feifeibear 8 months ago

#19 - polish benchmark flags

Pull Request - State: closed - Opened by feifeibear 8 months ago

#18 - revert benchmark flags

Pull Request - State: closed - Opened by feifeibear 8 months ago

#18 - revert benchmark flags

Pull Request - State: closed - Opened by feifeibear 8 months ago

#17 - add flags to benchmark script

Pull Request - State: closed - Opened by feifeibear 8 months ago

#17 - add flags to benchmark script

Pull Request - State: closed - Opened by feifeibear 8 months ago

#16 - remove useless print

Pull Request - State: closed - Opened by feifeibear 8 months ago

#16 - remove useless print

Pull Request - State: closed - Opened by feifeibear 8 months ago

#15 - long context attn qkvpacked

Pull Request - State: closed - Opened by feifeibear 8 months ago

#15 - long context attn qkvpacked

Pull Request - State: closed - Opened by feifeibear 8 months ago

#14 - add longctxattn benchmark

Pull Request - State: closed - Opened by feifeibear 8 months ago

#14 - add longctxattn benchmark

Pull Request - State: closed - Opened by feifeibear 8 months ago

#13 - add LongContextAttention

Pull Request - State: closed - Opened by feifeibear 8 months ago

#13 - add LongContextAttention

Pull Request - State: closed - Opened by feifeibear 8 months ago

#12 - fix process group init bug

Pull Request - State: closed - Opened by feifeibear 8 months ago

#11 - add use_ring_low_dim flag in hybrid test

Pull Request - State: closed - Opened by feifeibear 8 months ago

#10 - polish readme

Pull Request - State: closed - Opened by feifeibear 8 months ago

#9 - update readme for hybrid attention

Pull Request - State: closed - Opened by feifeibear 8 months ago

#8 - add hybrid sequence parallel

Pull Request - State: closed - Opened by feifeibear 8 months ago

#7 - polish ulysses attn test print

Pull Request - State: closed - Opened by feifeibear 8 months ago

#6 - check ulysses attention backward pass correctness

Pull Request - State: closed - Opened by feifeibear 8 months ago

#5 - check ulysses forward pass correctness

Pull Request - State: closed - Opened by feifeibear 8 months ago

#4 - fix a bug in all-to-all args and check ulysses output

Pull Request - State: closed - Opened by feifeibear 8 months ago

#3 - test torch mha version

Pull Request - State: closed - Opened by feifeibear 8 months ago

#2 - refactor 4D tensor all-to-all

Pull Request - State: closed - Opened by feifeibear 8 months ago

#1 - add ring flash attn as local attn

Pull Request - State: closed - Opened by feifeibear 8 months ago

GitHub / feifeibear/long-context-attention issues and pull requests