Language & LLMs

What Is Rotary Position Embedding?

Rotary position embedding (RoPE) encodes the position of tokens by applying a rotation to their query and key vectors based on their position. This method integrates position information directly into the attention computation. It is widely used in modern language models and helps them handle varying sequence lengths.