Mixture-of-Depths Attention
Scaling depth is a key driver for large language models (LLMs). Yet, as LLMs become deeper, they often suffer from signal degradation: informative features formed in shallow layers are gradually diluted by repeated residual updates, making them harder to recover in deeper layers. We introduce mixture-of-depths attention (MoDA), a mechanism that allows each attention head to attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers. We further describe a hardware-efficient algorithm for MoDA that resolves non-contiguous memory-access patterns, achieving 97.3% of FlashAttention-2's efficiency at a sequence length of 64K. Experiments on 1.5B-parameter models demonstrate that MoDA consistently outperforms strong baselines. Notably, it improves average perplexity by 0.2 across 10 validation benchmarks and increases average performance by 2.11% on 10 downstream tasks, with a negligible 3.7% FLOPs computational overhead. We also find that combining MoDA with post-norm yields better performance than using it with pre-norm. These results suggest that MoDA is a promising primitive for depth scaling. Code is released at https://github.com/hustvl/MoDA .
69b8fe396b3f36c249fbc579
Lianghui Zhu
admin_assigned
2026-03-17T09:59:01.230Z
643632c6a4bd75c62cc2d3b4
https://cdn-avatars.huggingface.co/v1/production/uploads/643632c6a4bd75c62cc2d3b4/rCqEb3_w54-QH2QSP2JhM.jpeg
Lianghui Zhu
LianghuiZhu
user
LianghuiZhu
69b8fe396b3f36c249fbc57a
Yuxin Fang
69b8fe396b3f36c249fbc57b
Bencheng Liao
admin_assigned
2026-03-17T09:59:12.410Z
6577073fc2bf55b1f6bafb49
/avatars/58803398b1a918b7570db17893e65122.svg
Bencheng liao
LegendBC
user
LegendBC
69b8fe396b3f36c249fbc57c
Shijie Wang
claimed_verified
2026-03-17T08:18:46.740Z
6472321922016353ae3ab2e9
/avatars/93d9e397ae6079ea0672d6b54234f388.svg
Shijie Wang
simonJJJ
user
simonJJJ
69b8fe396b3f36c249fbc57d
Tianheng Cheng
admin_assigned
2026-03-17T09:59:18.714Z
646b3db131968a60a01e4cf5
https://cdn-avatars.huggingface.co/v1/production/uploads/646b3db131968a60a01e4cf5/DhfdqUYQaD1Qa8Svw996J.jpeg
Tianheng Cheng
wondervictor
user
wondervictor
69b8fe396b3f36c249fbc57e
Zilong Huang
69b8fe396b3f36c249fbc57f
Chen Chen
69b8fe396b3f36c249fbc580
Lai Wei
69b8fe396b3f36c249fbc581
Yutao Zeng
69b8fe396b3f36c249fbc582
Ya Wang
69b8fe396b3f36c249fbc583
Yi Lin
69b8fe396b3f36c249fbc584
Yu Li
69b8fe396b3f36c249fbc585
Xinggang Wang
admin_assigned
2026-03-17T09:59:29.986Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
xinggangw
user
xinggangw
Mixture-of-Depths Attention
Scaling depth is a key driver for large language models (LLMs). Yet, as LLMs become deeper, they often suffer from signal degradation: informative features formed in shallow layers are gradually diluted by repeated residual updates, making them harder to recover in deeper layers. We introduce mixture-of-depths attention (MoDA), a mechanism that allows each attention head to attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers. We further describe a hardware-efficient algorithm for MoDA that resolves non-contiguous memory-access patterns, achieving 97.3% of FlashAttention-2's efficiency at a sequence length of 64K. Experiments on 1.5B-parameter models demonstrate that MoDA consistently outperforms strong baselines. Notably, it improves average perplexity by 0.2 across 10 validation benchmarks and increases average performance by 2.11% on 10 downstream tasks, with a negligible 3.7% FLOPs computational overhead. We also find that combining MoDA with post-norm yields better performance than using it with pre-norm. These results suggest that MoDA is a promising primitive for depth scaling. Code is released at https://github.com/hustvl/MoDA .