Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
Masked Diffusion Language Models (MDLMs) have recently emerged as a promising alternative to Autoregressive Language Models (ARLMs), leveraging a denoising objective that, in principle, should enable more uniform context utilisation. In this work, we examine the context comprehension abilities of MDLMs and uncover two key limitations. First, despite their more global training objective and bidirectional attention mechanism, similarly to ARLMS, MDLMs exhibit a strong locality bias: performance is highly sensitive to the position of relevant information within the input, favouring local over distant context. Second, we show that appending a large number of mask tokens--required for generation--can significantly degrade context comprehension. Through systematic ablations, we find that these masks act as distractors, reducing the model's ability to process relevant information. To address this, we introduce a mask-agnostic loss function that encourages predictions to remain invariant to the number of appended masks. Fine-tuning with this objective substantially mitigates the distracting effect of masks, improving robustness of MDLMs. Overall, our findings reveal critical limitations of the current MDLM training paradigm and provide actionable insights for building diffusion-based language models with stronger context comprehension.
692d866d0ce2ec9461c880bb
Julianna Piskorz
claimed_verified
2025-12-01T16:24:36.550Z
67d44d099c024761ad196132
/avatars/614c79d727424e427d65fe6ee7b53abd.svg
Julianna Piskorz
jpiskorz
user
jpiskorz
692d866d0ce2ec9461c880bc
Cristina Pinneri
692d866d0ce2ec9461c880bd
Alvaro Correia
692d866d0ce2ec9461c880be
Motasem Alfarra
692d866d0ce2ec9461c880bf
Risheek Garrepalli
692d866d0ce2ec9461c880c0
Christos Louizos
Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
Masked Diffusion Language Models (MDLMs) have recently emerged as a promising alternative to Autoregressive Language Models (ARLMs), leveraging a denoising objective that, in principle, should enable more uniform context utilisation. In this work, we examine the context comprehension abilities of MDLMs and uncover two key limitations. First, despite their more global training objective and bidirectional attention mechanism, similarly to ARLMS, MDLMs exhibit a strong locality bias: performance is highly sensitive to the position of relevant information within the input, favouring local over distant context. Second, we show that appending a large number of mask tokens--required for generation--can significantly degrade context comprehension. Through systematic ablations, we find that these masks act as distractors, reducing the model's ability to process relevant information. To address this, we introduce a mask-agnostic loss function that encourages predictions to remain invariant to the number of appended masks. Fine-tuning with this objective substantially mitigates the distracting effect of masks, improving robustness of MDLMs. Overall, our findings reveal critical limitations of the current MDLM training paradigm and provide actionable insights for building diffusion-based language models with stronger context comprehension.
Masked Diffusion Language Models (MDLMs) exhibit locality bias and are negatively impacted by appended mask tokens, but a mask-agnostic loss function improves their context comprehension.
Masked Diffusion Language Models MDLMs Autoregressive Language Models ARLMs denoising objective bidirectional attention locality bias mask tokens mask-agnostic loss function context comprehension