Mitigating Attention Sinks and Massive Activations in Audio-Visual
  Speech Recognition with LLMS
Large language models (LLMs) have recently advanced auditory speech
recognition (ASR), visual speech recognition (VSR), and audio-visual speech
recognition (AVSR). However, understanding of their internal dynamics under
fine-tuning remains limited. In natural language processing, recent work has
revealed attention sinks, tokens that attract disproportionately high
attention, and associated massive activations in which some features of sink
tokens exhibit huge activation in LLMs. In this work, we are the first to study
these phenomena in multimodal speech recognition. Through a detailed analysis
of audio-visual LLMs, we identify attention sinks and massive activations not
only at the BOS token but also at intermediate low-semantic tokens across ASR,
VSR, and AVSR. We show that massive activations originate in the MLP layers and
correspond to fixed feature indices across all sink tokens. We further show
that intermediate sink tokens exhibit high cosine similarity to the BOS token,
thereby amplifying attention and activation. Building on these insights, we
introduce a simple decorrelation loss that reduces cosine similarity between
BOS and other tokens, effectively mitigating intermediate sinks and massive
activations. Furthermore, our method improves word error rate (WER) under high
audio-visual feature downsampling while remaining stable at lower downsampling
rates.
69007e71646208eac0d1ef2a
Anand
69007e71646208eac0d1ef2b
Umberto Cappellazzo
claimed_verified
2025-10-28T15:26:41.351Z
64903f017b630c141867877f
https://cdn-avatars.huggingface.co/v1/production/uploads/64903f017b630c141867877f/IvVHLmY95_3J7Jd-E5Lfi.jpeg
Umberto Cappellazzo
hisoka94
user
hisoka94
69007e71646208eac0d1ef2c
Stavros Petridis
69007e71646208eac0d1ef2d
Maja Pantic
Mitigating Attention Sinks and Massive Activations in Audio-Visual
  Speech Recognition with LLMS
Large language models (LLMs) have recently advanced auditory speech
recognition (ASR), visual speech recognition (VSR), and audio-visual speech
recognition (AVSR). However, understanding of their internal dynamics under
fine-tuning remains limited. In natural language processing, recent work has
revealed attention sinks, tokens that attract disproportionately high
attention, and associated massive activations in which some features of sink
tokens exhibit huge activation in LLMs. In this work, we are the first to study
these phenomena in multimodal speech recognition. Through a detailed analysis
of audio-visual LLMs, we identify attention sinks and massive activations not
only at the BOS token but also at intermediate low-semantic tokens across ASR,
VSR, and AVSR. We show that massive activations originate in the MLP layers and
correspond to fixed feature indices across all sink tokens. We further show
that intermediate sink tokens exhibit high cosine similarity to the BOS token,
thereby amplifying attention and activation. Building on these insights, we
introduce a simple decorrelation loss that reduces cosine similarity between
BOS and other tokens, effectively mitigating intermediate sinks and massive
activations. Furthermore, our method improves word error rate (WER) under high
audio-visual feature downsampling while remaining stable at lower downsampling
rates.
Attention sinks and massive activations in multimodal speech recognition are identified and mitigated using a decorrelation loss, improving word error rate under high feature downsampling.
attention sinks massive activations BOS token MLP layers cosine similarity decorrelation loss word error rate audio-visual feature downsampling