Attention Residuals
Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, Block AttnRes becomes a practical drop-in replacement for standard residual connections with minimal overhead.
  Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks.
69b8d6866b3f36c249fbc493
Kimi Team
69b8d6866b3f36c249fbc494
Guangyu Chen
69b8d6866b3f36c249fbc495
Yu Zhang
claimed_verified
2026-03-17T17:34:43.408Z
638f5839f6de4b9e7e1627fb
https://cdn-avatars.huggingface.co/v1/production/uploads/638f5839f6de4b9e7e1627fb/6QGkrqRag6-GnH9k60Oil.jpeg
Yu Zhang
yzhangcs
user
yzhangcs
69b8d6866b3f36c249fbc496
Jianlin Su
69b8d6866b3f36c249fbc497
Weixin Xu
69b8d6866b3f36c249fbc498
Siyuan Pan
69b8d6866b3f36c249fbc499
Yaoyu Wang
69b8d6866b3f36c249fbc49a
Yucheng Wang
69b8d6866b3f36c249fbc49b
Guanduo Chen
69b8d6866b3f36c249fbc49c
Bohong Yin
69b8d6866b3f36c249fbc49d
Yutian Chen
69b8d6866b3f36c249fbc49e
Junjie Yan
69b8d6866b3f36c249fbc49f
Ming Wei
69b8d6866b3f36c249fbc4a0
Y. Zhang
69b8d6866b3f36c249fbc4a1
Fanqing Meng
69b8d6866b3f36c249fbc4a2
Chao Hong
69b8d6866b3f36c249fbc4a3
Xiaotong Xie
69b8d6866b3f36c249fbc4a4
Shaowei Liu
69b8d6866b3f36c249fbc4a5
Enzhe Lu
69b8d6866b3f36c249fbc4a6
Yunpeng Tai
69b8d6866b3f36c249fbc4a7
Yanru Chen
69b8d6866b3f36c249fbc4a8
Xin Men
69b8d6866b3f36c249fbc4a9
Haiqing Guo
69b8d6866b3f36c249fbc4aa
Y. Charles
69b8d6866b3f36c249fbc4ab
Haoyu Lu
69b8d6866b3f36c249fbc4ac
Lin Sui
69b8d6866b3f36c249fbc4ad
Jinguo Zhu
69b8d6866b3f36c249fbc4ae
Zaida Zhou
69b8d6866b3f36c249fbc4af
Weiran He
69b8d6866b3f36c249fbc4b0
Weixiao Huang
69b8d6866b3f36c249fbc4b1
Xinran Xu
69b8d6866b3f36c249fbc4b2
Yuzhi Wang
69b8d6866b3f36c249fbc4b3
Guokun Lai
69b8d6866b3f36c249fbc4b4
Yulun Du
69b8d6866b3f36c249fbc4b5
Yuxin Wu
69b8d6866b3f36c249fbc4b6
Zhilin Yang
69b8d6866b3f36c249fbc4b7
Xinyu Zhou
Attention Residuals
Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, Block AttnRes becomes a practical drop-in replacement for standard residual connections with minimal overhead.
  Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks.