Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in Large Language Models (LLMs) by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score calculation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re-incorporates this discarded imaginary component. Our method leverages the full complex-valued representation to create a dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling of long-context dependencies by preserving more positional information. Furthermore, evaluations on a suite of long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant as context length increases. The code is available at https://github.com/OpenMOSS/rope_pp.
693794d319d912300c34a291
Xiaoran Liu
claimed_verified
2025-12-09T09:38:01.257Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Xiaoran Liu (SII)
SII-xrliu
user
SII-xrliu
693794d319d912300c34a292
Yuerong Song
693794d319d912300c34a293
Zhigeng Liu
693794d319d912300c34a294
Zengfeng Huang
admin_assigned
2025-12-09T10:35:11.754Z
64805e6dde559d48dbb00627
/avatars/29ca34546411dcc28bbc934e3c26a2ba.svg
Zengfeng
ZengfengHuang
user
ZengfengHuang
693794d319d912300c34a295
Qipeng Guo
admin_assigned
2025-12-09T10:35:18.117Z
6491cd52b1e5d3444528edb1
/avatars/a85635d886c7f157b6723dec5c01c030.svg
Qipeng Guo
QipengGuo
user
QipengGuo
693794d319d912300c34a296
Zhaoxiang Liu
693794d319d912300c34a297
Shiguo Lian
693794d319d912300c34a298
Ziwei He
claimed_verified
2025-12-09T20:17:41.271Z
64de18f41d826d7355c285e7
/avatars/23e2c44e3a593415becc02463980f6e8.svg
Ziwei He
ziweihe
user
ziweihe
693794d319d912300c34a299
Xipeng Qiu
admin_assigned
2025-12-09T10:35:28.411Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
xpqiu
user
xpqiu
Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in Large Language Models (LLMs) by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score calculation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re-incorporates this discarded imaginary component. Our method leverages the full complex-valued representation to create a dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling of long-context dependencies by preserving more positional information. Furthermore, evaluations on a suite of long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant as context length increases. The code is available at https://github.com/OpenMOSS/rope_pp.
The paper proposes a method to enhance Rotary Position Embeddings by utilizing both the real and imaginary components of the complex-valued dot product, improving long-context modeling in Large Language Models.
Rotary Position Embeddings Large Language Models complex-valued dot product attention score long-context dependencies positional information