Balancing Understanding and Generation in Discrete Diffusion Models
In discrete generative modeling, two dominant paradigms demonstrate divergent capabilities: Masked Diffusion Language Models (MDLM) excel at semantic understanding and zero-shot generalization, whereas Uniform-noise Diffusion Language Models (UDLM) achieve strong few-step generation quality, yet neither attains balanced performance across both dimensions. To address this, we propose XDLM, which bridges the two paradigms via a stationary noise kernel. XDLM offers two key contributions: (1) it provides a principled theoretical unification of MDLM and UDLM, recovering each paradigm as a special case; and (2) an alleviated memory bottleneck enabled by an algebraic simplification of the posterior probabilities. Experiments demonstrate that XDLM advances the Pareto frontier between understanding capability and generation quality. Quantitatively, XDLM surpasses UDLM by 5.4 points on zero-shot text benchmarks and outperforms MDLM in few-step image generation (FID 54.1 vs. 80.8). When scaled to tune an 8B-parameter large language model, XDLM achieves 15.0 MBPP in just 32 steps, effectively doubling the baseline performance. Finally, analysis of training dynamics reveals XDLM's superior potential for long-term scaling. Code is available at https://github.com/MzeroMiko/XDLM
69829ac79084cb4f0ecb55e4
Yue Liu
claimed_verified
2026-02-04T12:29:22.643Z
65aa76b1cb5b4fb08ecb087c
https://cdn-avatars.huggingface.co/v1/production/uploads/65aa76b1cb5b4fb08ecb087c/0_x6Zb-H36XfugcsQ80Zh.jpeg
Liu Yue
Mzero17
user
Mzero17
69829ac79084cb4f0ecb55e5
Yuzhong Zhao
claimed_verified
2026-02-04T12:29:19.801Z
635b6b3590204c509f9755bd
/avatars/94398dd17ca1a2e74dc8d775abf636a7.svg
zhaoyuzhong
callsys
user
callsys
69829ac79084cb4f0ecb55e6
Zheyong Xie
69829ac79084cb4f0ecb55e7
Qixiang Ye
69829ac79084cb4f0ecb55e8
Jianbin Jiao
69829ac79084cb4f0ecb55e9
Yao Hu
69829ac79084cb4f0ecb55ea
Shaosheng Cao
69829ac79084cb4f0ecb55eb
Yunfan Liu
Balancing Understanding and Generation in Discrete Diffusion Models
In discrete generative modeling, two dominant paradigms demonstrate divergent capabilities: Masked Diffusion Language Models (MDLM) excel at semantic understanding and zero-shot generalization, whereas Uniform-noise Diffusion Language Models (UDLM) achieve strong few-step generation quality, yet neither attains balanced performance across both dimensions. To address this, we propose XDLM, which bridges the two paradigms via a stationary noise kernel. XDLM offers two key contributions: (1) it provides a principled theoretical unification of MDLM and UDLM, recovering each paradigm as a special case; and (2) an alleviated memory bottleneck enabled by an algebraic simplification of the posterior probabilities. Experiments demonstrate that XDLM advances the Pareto frontier between understanding capability and generation quality. Quantitatively, XDLM surpasses UDLM by 5.4 points on zero-shot text benchmarks and outperforms MDLM in few-step image generation (FID 54.1 vs. 80.8). When scaled to tune an 8B-parameter large language model, XDLM achieves 15.0 MBPP in just 32 steps, effectively doubling the baseline performance. Finally, analysis of training dynamics reveals XDLM's superior potential for long-term scaling. Code is available at https://github.com/MzeroMiko/XDLM
XDLM unifies Masked Diffusion Language Models and Uniform-noise Diffusion Language Models through a stationary noise kernel, achieving improved performance in both semantic understanding and generation quality.
Masked Diffusion Language Models Uniform-noise Diffusion Language Models stationary noise kernel Pareto frontier posterior probabilities algebraic simplification large language model FID MBPP training dynamics