Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis
Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations from a temporally continuous perspective. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension. Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at https://github.com/MIRALab-USTC/LLM-TAPPA.
6980623f6676f93322706710
Qingyue Yang
6980623f6676f93322706711
Jie Wang
6980623f6676f93322706712
Xing Li
6980623f6676f93322706713
Yinqi Bai
6980623f6676f93322706714
Xialiang Tong
6980623f6676f93322706715
Huiling Zhen
6980623f6676f93322706716
Jianye Hao
6980623f6676f93322706717
Mingxuan Yuan
6980623f6676f93322706718
Bin Li
Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis
Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations from a temporally continuous perspective. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension. Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at https://github.com/MIRALab-USTC/LLM-TAPPA.
Temporal Attention Pattern Predictability Analysis (TAPPA) provides a unified framework for understanding attention patterns in large language models by analyzing their mathematical formulations from a temporal perspective, distinguishing predictable from unpredictable patterns based on query self-similarity.
Attention patterns large language models Temporal Attention Pattern Predictability Analysis TAPPA query self-similarity Rotary Positional Embeddings RoPE KV cache compression LLM pruning