Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persistent challenge across various Transformers is Attention Sink (AS), in which a disproportionate amount of attention is focused on a small subset of specific yet uninformative tokens. AS complicates interpretability, significantly affecting the training and inference dynamics, and exacerbates issues such as hallucinations. In recent years, substantial research has been dedicated to understanding and harnessing AS. However, a comprehensive survey that systematically consolidates AS-related research and offers guidance for future advancements remains lacking. To address this gap, we present the first survey on AS, structured around three key dimensions that define the current research landscape: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Our work provides a pivotal contribution by clarifying key concepts and guiding researchers through the evolution and trends of the field. We envision this survey as a definitive resource, empowering researchers and practitioners to effectively manage AS within the current Transformer paradigm, while simultaneously inspiring innovative advancements for the next generation of Transformers. The paper list of this work is available at https://github.com/ZunhaiSu/Awesome-Attention-Sink.
69ddaeadaea31e227c22c504
Zunhai Su
claimed_verified
2026-04-14T08:15:27.744Z
655b4f5cc11dee7f7e882a0c
https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png
zunhaisu
zunhai
user
zunhai
69ddaeadaea31e227c22c505
Hengyuan Zhang
69ddaeadaea31e227c22c506
Wei Wu
69ddaeadaea31e227c22c507
Yifan Zhang
69ddaeadaea31e227c22c508
Yaxiu Liu
69ddaeadaea31e227c22c509
He Xiao
69ddaeadaea31e227c22c50a
Qingyao Yang
69ddaeadaea31e227c22c50b
Yuxuan Sun
69ddaeadaea31e227c22c50c
Rui Yang
69ddaeadaea31e227c22c50d
Chao Zhang
69ddaeadaea31e227c22c50e
Keyu Fan
69ddaeadaea31e227c22c50f
Weihao Ye
69ddaeadaea31e227c22c510
Jing Xiong
69ddaeadaea31e227c22c511
Hui Shen
69ddaeadaea31e227c22c512
Chaofan Tao
69ddaeadaea31e227c22c513
Taiqiang Wu
claimed_verified
2026-04-15T12:07:53.964Z
6621cea88850e38ffbb1854f
https://cdn-avatars.huggingface.co/v1/production/uploads/6621cea88850e38ffbb1854f/qmKHts7YCUylxqzPHk4uT.jpeg
Taki WU
taki555
user
taki555
69ddaeadaea31e227c22c514
Zhongwei Wan
69ddaeadaea31e227c22c515
Yulei Qian
69ddaeadaea31e227c22c516
Yuchen Xie
69ddaeadaea31e227c22c517
Ngai Wong
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persistent challenge across various Transformers is Attention Sink (AS), in which a disproportionate amount of attention is focused on a small subset of specific yet uninformative tokens. AS complicates interpretability, significantly affecting the training and inference dynamics, and exacerbates issues such as hallucinations. In recent years, substantial research has been dedicated to understanding and harnessing AS. However, a comprehensive survey that systematically consolidates AS-related research and offers guidance for future advancements remains lacking. To address this gap, we present the first survey on AS, structured around three key dimensions that define the current research landscape: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Our work provides a pivotal contribution by clarifying key concepts and guiding researchers through the evolution and trends of the field. We envision this survey as a definitive resource, empowering researchers and practitioners to effectively manage AS within the current Transformer paradigm, while simultaneously inspiring innovative advancements for the next generation of Transformers. The paper list of this work is available at https://github.com/ZunhaiSu/Awesome-Attention-Sink.
Transformers face challenges from Attention Sink phenomenon where excessive attention focuses on uninformative tokens, impacting interpretability and performance, necessitating comprehensive research survey addressing fundamental usage, mechanistic understanding, and strategic mitigation approaches.
Transformers Attention Sink attention mechanisms interpretability training dynamics inference dynamics hallucinations mechanistic interpretation strategic mitigation