Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models
Diffusion large language models (dLLMs) theoretically permit token decoding in arbitrary order, a flexibility that could enable richer exploration of reasoning paths than autoregressive (AR) LLMs. In practice, however, random-order decoding often hurts generation quality. To mitigate this, low-confidence remasking improves single-sample quality (e.g., Pass@1) by prioritizing confident tokens, but it also suppresses exploration and limits multi-sample gains (e.g., Pass@k), creating a fundamental quality--exploration dilemma. In this paper, we provide a unified explanation of this dilemma. We show that low-confidence remasking improves a myopic proxy for quality while provably constraining the entropy of the induced sequence distribution. To overcome this limitation, we characterize the optimal distribution that explicitly balances quality and exploration, and develop a simple Independent Metropolis--Hastings sampler that approximately targets this distribution during decoding. Experiments across a range of reasoning benchmarks including MATH500, AIME24/25, HumanEval, and MBPP show that our approach yields better exploration-quality tradeoff than both random and low-confidence remasking.
69d5389263f089495f06b2d3
Liancheng Fang
69d5389263f089495f06b2d4
Aiwei Liu
69d5389263f089495f06b2d5
Henry Peng Zou
69d5389263f089495f06b2d6
Yankai Chen
69d5389263f089495f06b2d7
Enze Ma
69d5389263f089495f06b2d8
Leyi Pan
69d5389263f089495f06b2d9
Chunyu Miao
69d5389263f089495f06b2da
Wei-Chieh Huang
69d5389263f089495f06b2db
Xue Liu
69d5389263f089495f06b2dc
Philip S. Yu
Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models
Diffusion large language models (dLLMs) theoretically permit token decoding in arbitrary order, a flexibility that could enable richer exploration of reasoning paths than autoregressive (AR) LLMs. In practice, however, random-order decoding often hurts generation quality. To mitigate this, low-confidence remasking improves single-sample quality (e.g., Pass@1) by prioritizing confident tokens, but it also suppresses exploration and limits multi-sample gains (e.g., Pass@k), creating a fundamental quality--exploration dilemma. In this paper, we provide a unified explanation of this dilemma. We show that low-confidence remasking improves a myopic proxy for quality while provably constraining the entropy of the induced sequence distribution. To overcome this limitation, we characterize the optimal distribution that explicitly balances quality and exploration, and develop a simple Independent Metropolis--Hastings sampler that approximately targets this distribution during decoding. Experiments across a range of reasoning benchmarks including MATH500, AIME24/25, HumanEval, and MBPP show that our approach yields better exploration-quality tradeoff than both random and low-confidence remasking.
Diffusion large language models face a tradeoff between generation quality and reasoning path exploration, which is addressed through a novel sampling method that balances these competing objectives.
diffusion large language models autoregressive LLMs token decoding random-order decoding low-confidence remasking Pass@$1 Pass@$k myopic proxy entropy sequence distribution Independent Metropolis--Hastings sampler reasoning benchmarks MATH500 AIME24/25 HumanEval MBPP