Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
One of the most compelling features of global discrete diffusion language models is their global bidirectional contextual capability. However, existing block-based diffusion studies tend to introduce autoregressive priors, which, while offering benefits, can cause models to lose this global coherence at the macro level. To regain global contextual understanding while preserving the advantages of the semi-autoregressive paradigm, we propose Diffusion in Diffusion, a 'draft-then-refine' framework designed to overcome the irreversibility and myopia problems inherent in block diffusion models. Our approach first employs block diffusion to generate rapid drafts using small blocks, then refines these drafts through global bidirectional diffusion with a larger bidirectional receptive field. We utilize snapshot confidence remasking to identify the most critical tokens that require modification, and apply mix-scale training to expand the block diffusion model's global capabilities. Empirical results demonstrate that our approach sets a new benchmark for discrete diffusion models on the OpenWebText dataset. Using only 26% of the fine-tuning budget of baseline models, we reduce generative perplexity from 25.7 to 21.9, significantly narrowing the performance gap with autoregressive models.
69785d8f026bdf0473116f8d
Linrui Ma
69785d8f026bdf0473116f8e
Yufei Cui
69785d8f026bdf0473116f8f
Kai Han
admin_assigned
2026-01-27T14:38:15.388Z
65e52e7d27dc8aa470a640e3
/avatars/022a179d14de29b9ab9d96fcc85aa264.svg
hankai
hankaixyz
user
hankaixyz
69785d8f026bdf0473116f90
Yunhe Wang
Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
One of the most compelling features of global discrete diffusion language models is their global bidirectional contextual capability. However, existing block-based diffusion studies tend to introduce autoregressive priors, which, while offering benefits, can cause models to lose this global coherence at the macro level. To regain global contextual understanding while preserving the advantages of the semi-autoregressive paradigm, we propose Diffusion in Diffusion, a 'draft-then-refine' framework designed to overcome the irreversibility and myopia problems inherent in block diffusion models. Our approach first employs block diffusion to generate rapid drafts using small blocks, then refines these drafts through global bidirectional diffusion with a larger bidirectional receptive field. We utilize snapshot confidence remasking to identify the most critical tokens that require modification, and apply mix-scale training to expand the block diffusion model's global capabilities. Empirical results demonstrate that our approach sets a new benchmark for discrete diffusion models on the OpenWebText dataset. Using only 26% of the fine-tuning budget of baseline models, we reduce generative perplexity from 25.7 to 21.9, significantly narrowing the performance gap with autoregressive models.
A 'draft-then-refine' framework for discrete diffusion language models that recovers global contextual understanding while maintaining semi-autoregressive efficiency through block diffusion and global bidirectional processing.
discrete diffusion language models global bidirectional contextual capability block-based diffusion autoregressive priors semi-autoregressive paradigm diffusion in diffusion draft-then-refine framework irreversibility myopia snapshot confidence remasking mix-scale training generative perplexity