Knocking-Heads Attention
Multi-head attention (MHA) has become the cornerstone of modern large
language models, enhancing representational capacity through parallel attention
heads. However, increasing the number of heads inherently weakens individual
head capacity, and existing attention mechanisms - whether standard MHA or its
variants like grouped-query attention (GQA) and grouped-tied attention (GTA) -
simply concatenate outputs from isolated heads without strong interaction. To
address this limitation, we propose knocking-heads attention (KHA), which
enables attention heads to "knock" on each other - facilitating cross-head
feature-level interactions before the scaled dot-product attention. This is
achieved by applying a shared, diagonally-initialized projection matrix across
all heads. The diagonal initialization preserves head-specific specialization
at the start of training while allowing the model to progressively learn
integrated cross-head representations. KHA adds only minimal parameters and
FLOPs and can be seamlessly integrated into MHA, GQA, GTA, and other attention
variants. We validate KHA by training a 6.1B parameter MoE model (1.01B
activated) on 1T high-quality tokens. Compared to baseline attention
mechanisms, KHA brings superior and more stable training dynamics, achieving
better performance across downstream tasks.
69003b6822d452aac6dd43b8
Zhanchao Zhou
claimed_verified
2025-10-28T15:36:08.904Z
671b3d82101c0d9f78749256
/avatars/902c1006280c6552885b0b946cd2bb16.svg
Zhanchao Zhou
Zcchill
user
Zcchill
69003b6822d452aac6dd43b9
Xiaodong Chen
claimed_verified
2025-10-29T12:49:55.484Z
67be7cd21616162fc336cb44
/avatars/e58cc3c2d1484419222a5ccfc11f5c48.svg
Xiaodong Chen
XiaodongChen
user
XiaodongChen
69003b6822d452aac6dd43ba
Haoxing Chen
claimed_verified
2025-10-28T15:36:11.263Z
63898c562a897944ea5f07a2
https://cdn-avatars.huggingface.co/v1/production/uploads/63898c562a897944ea5f07a2/oG1ia44ICpAk1tReIaX0G.jpeg
Haoxing chen
HaoxingChen
user
HaoxingChen
69003b6822d452aac6dd43bb
Zhenzhong Lan
69003b6822d452aac6dd43bc
Jianguo Li
claimed_verified
2025-12-25T20:54:35.985Z
6539fc3fe279d2fc80c88902
/avatars/432ac2287a7f7d04c2c7e505070a3331.svg
Jianguo Li
jglee2046
user
jglee2046
Knocking-Heads Attention
Multi-head attention (MHA) has become the cornerstone of modern large
language models, enhancing representational capacity through parallel attention
heads. However, increasing the number of heads inherently weakens individual
head capacity, and existing attention mechanisms - whether standard MHA or its
variants like grouped-query attention (GQA) and grouped-tied attention (GTA) -
simply concatenate outputs from isolated heads without strong interaction. To
address this limitation, we propose knocking-heads attention (KHA), which
enables attention heads to "knock" on each other - facilitating cross-head
feature-level interactions before the scaled dot-product attention. This is
achieved by applying a shared, diagonally-initialized projection matrix across
all heads. The diagonal initialization preserves head-specific specialization
at the start of training while allowing the model to progressively learn
integrated cross-head representations. KHA adds only minimal parameters and
FLOPs and can be seamlessly integrated into MHA, GQA, GTA, and other attention
variants. We validate KHA by training a 6.1B parameter MoE model (1.01B
activated) on 1T high-quality tokens. Compared to baseline attention
mechanisms, KHA brings superior and more stable training dynamics, achieving
better performance across downstream tasks.
Knocking-heads attention (KHA) enhances multi-head attention by enabling cross-head interactions, improving training dynamics and performance in large language models.
multi-head attention MHA grouped-query attention GQA grouped-tied attention GTA knocking-heads attention KHA scaled dot-product attention parameter-efficient MoE high-quality tokens training dynamics downstream tasks