DoPE: Denoising Rotary Position Embedding
Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length extrapolation. We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional Encoding (DoPE), a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map. Leveraging the noise characteristics of the feature map, we further reparameterize it with a parameter-free Gaussian distribution to achieve robust extrapolation. Our method theoretically reveals the underlying cause of the attention sink phenomenon and its connection to truncated matrix entropy. Experiments on needle-in-a-haystack and many-shot in-context learning tasks demonstrate that DoPE significantly improves retrieval accuracy and reasoning stability across extended contexts (up to 64K tokens). The results show that the denoising strategy for positional embeddings effectively mitigates attention sinks and restores balanced attention patterns, providing a simple yet powerful solution for improving length generalization. Our project page is Project: https://The-physical-picture-of-LLMs.github.io
6917090db63bfc66e049897a
Jing Xiong
6917090db63bfc66e049897b
Liyang Fan
6917090db63bfc66e049897c
Hui Shen
claimed_verified
2026-01-13T08:25:03.334Z
640feb91b0ee289c8583dd59
https://cdn-avatars.huggingface.co/v1/production/uploads/640feb91b0ee289c8583dd59/CA4UZj2mEsJO2UR0NnN5H.jpeg
Hui Shen
Cloudriver
user
Cloudriver
6917090db63bfc66e049897d
Zunhai Su
claimed_verified
2025-11-18T14:09:16.218Z
655b4f5cc11dee7f7e882a0c
https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png
zunhaisu
zunhai
user
zunhai
6917090db63bfc66e049897e
Min Yang
6917090db63bfc66e049897f
Lingpeng Kong
6917090db63bfc66e0498980
Ngai Wong
DoPE: Denoising Rotary Position Embedding
Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length extrapolation. We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional Encoding (DoPE), a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map. Leveraging the noise characteristics of the feature map, we further reparameterize it with a parameter-free Gaussian distribution to achieve robust extrapolation. Our method theoretically reveals the underlying cause of the attention sink phenomenon and its connection to truncated matrix entropy. Experiments on needle-in-a-haystack and many-shot in-context learning tasks demonstrate that DoPE significantly improves retrieval accuracy and reasoning stability across extended contexts (up to 64K tokens). The results show that the denoising strategy for positional embeddings effectively mitigates attention sinks and restores balanced attention patterns, providing a simple yet powerful solution for improving length generalization. Our project page is Project: https://The-physical-picture-of-LLMs.github.io
Denoising Positional Encoding (DoPE) enhances length generalization in Transformer models by detecting and mitigating noisy frequency bands in positional embeddings, improving retrieval accuracy and reasoning stability.
Rotary Position Embedding (RoPE) Transformer models attention map positional encoding Denoising Positional Encoding (DoPE) truncated matrix entropy Gaussian distribution attention sink phenomenon needle-in-a-haystack many-shot in-context learning retrieval accuracy reasoning stability length generalization