lucidrains/rotary-embedding-torch issues and pull requests

#40 - How to use in stream inference?

Issue - State: open - Opened by haha010508 2 months ago

#39 - Was inspired by the paper you posted but i think it can be taken further by expanding the idea to multiple attention types (not unlike the paper)

Issue - State: open - Opened by sine2pi 3 months ago

#38 - Change cached seq_len to int to enable compilation

Pull Request - State: closed - Opened by f0k 3 months ago - 1 comment

#37 - Unused params if cache_if_possible=True when multiple rotary dimensions are used

Issue - State: closed - Opened by lukasschmit 4 months ago - 1 comment

#36 - Slower than absolute positional embeddings?

Issue - State: open - Opened by umarbutler 5 months ago - 4 comments

#35 - Explicit casting instead of autocasting

Issue - State: open - Opened by lminer 5 months ago

#34 - Fix chi scale multiplication

Pull Request - State: closed - Opened by TimFelixBeyer 6 months ago - 1 comment

#33 - Add sequence position interpolation to axial RoPE

Pull Request - State: closed - Opened by tasansal 6 months ago - 1 comment

#32 - Fine-tuning Axial RoPE with frequency scaling?

Issue - State: open - Opened by tasansal 6 months ago

#31 - apply_rotary_emb - remove inplace operation

Pull Request - State: closed - Opened by blasscoc 6 months ago - 1 comment

#30 - RoPE embeddings

Issue - State: open - Opened by PRamoneda 6 months ago - 1 comment

#29 - Request for YaRN

Issue - State: open - Opened by VarunGumma 7 months ago

#28 - Lastest commit incompatible with local_attention

Issue - State: closed - Opened by MarcusLoppe 8 months ago - 3 comments

#27 - xPOS embeddings during inference

Issue - State: closed - Opened by VarunGumma 8 months ago - 2 comments

#26 - LieRE: Generalizing Rotary Position Encodings. Beats RoPE-mixed by large margin and is much faster (compute-wise)

Issue - State: closed - Opened by kabachuha 8 months ago - 34 comments

#25 - RoPE-Mixed: Improvement over Axial for n-D

Issue - State: open - Opened by tasansal 8 months ago - 1 comment

#24 - nan loss when training in fp8 with transformer engine

Issue - State: closed - Opened by saurabh-kataria 8 months ago - 1 comment

#23 - Repeat order.

Issue - State: closed - Opened by AliYoussef97 10 months ago

#22 - added indexing

Pull Request - State: closed - Opened by AlxSp 12 months ago

#21 - Request for permission to publish a Rust port of this python module

Issue - State: closed - Opened by Mekadrom 12 months ago - 1 comment

#20 - `torch.cat` failes in `apply_rotary_emb` when `freqs.shape[-1] == t.shape[-1]`, and `start_index = 0`

Issue - State: open - Opened by mattaltberg 12 months ago - 1 comment

#19 - RoPE on Images

Issue - State: closed - Opened by aaprasad about 1 year ago - 1 comment

#18 - caching frequency results in RuntimeError: Trying to backward through the graph a second time

Issue - State: closed - Opened by wren93 about 1 year ago - 3 comments

#17 - Is 'broadcat' part of the API?

Issue - State: closed - Opened by rsxdalv about 1 year ago - 5 comments

#16 - Error caused by tensor-type seq_len

Issue - State: closed - Opened by cmunna0052 about 1 year ago - 1 comment

#15 - Model hangs on eval

Issue - State: open - Opened by GarrettMerz about 1 year ago - 18 comments

#14 - implementing on vision tranformers

Issue - State: closed - Opened by mukvnd over 1 year ago - 2 comments

#13 - Does this library support 2D RoPE embeddings?

Issue - State: closed - Opened by logicchains over 1 year ago - 2 comments

#12 - Support for sequence length ordering

Issue - State: closed - Opened by iiSeymour over 1 year ago - 7 comments

#11 - Bug in cache

Issue - State: closed - Opened by N0r9st over 1 year ago - 1 comment

#10 - Usage with x-transformers

Issue - State: open - Opened by sonovice over 1 year ago - 4 comments

#9 - Bfloat16 support for use_xpos=True

Pull Request - State: closed - Opened by rostro36 over 1 year ago - 8 comments

#8 - Using with xpos causes NaNs after rotating Q, K

Issue - State: closed - Opened by andersonbcdefg almost 2 years ago - 2 comments

#7 - AttributeError: 'NoneType' object has no attribute 'to'

Issue - State: closed - Opened by yingzhao27 almost 2 years ago - 3 comments

#6 - freqs reference

Pull Request - State: open - Opened by biirving almost 2 years ago

#5 - Tricks for training with RoPE? Specific initialisers for QK projections?

Issue - State: open - Opened by thorinf almost 2 years ago

#4 - why dim of q be different from dim of RotaryEmbedding

Issue - State: open - Opened by HiSultryMan almost 2 years ago - 2 comments

#3 - Length Extrapolatable Rotary Embeddings

Issue - State: open - Opened by hugofloresgarcia almost 2 years ago - 2 comments

#2 - Custom position offset when rotating queries or keys

Pull Request - State: closed - Opened by krasserm about 2 years ago - 1 comment

#1 - about axial rotary embeddings

Issue - State: open - Opened by raindrop313 over 2 years ago

GitHub / lucidrains/rotary-embedding-torch issues and pull requests