Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive
  Experts
Diffusion-based large language models (dLLMs) are trained flexibly to model
extreme dependence in the data distribution; however, how to best utilize this
information at inference time remains an open problem. In this work, we uncover
an interesting property of these models: dLLMs trained on textual data
implicitly learn a mixture of semi-autoregressive experts, where different
generation orders reveal different specialized behaviors. We show that
committing to any single, fixed inference time schedule, a common practice,
collapses performance by failing to leverage this latent ensemble. To address
this, we introduce HEX (Hidden semiautoregressive EXperts for test-time
scaling), a training-free inference method that ensembles across heterogeneous
block schedules. By doing a majority vote over diverse block-sized generation
paths, HEX robustly avoids failure modes associated with any single fixed
schedule. On reasoning benchmarks such as GSM8K, it boosts accuracy by up to
3.56X (from 24.72% to 88.10%), outperforming top-K margin inference and
specialized fine-tuned methods like GRPO, without additional training. HEX even
yields significant gains on MATH benchmark from 16.40% to 40.00%, scientific
reasoning on ARC-C from 54.18% to 87.80%, and TruthfulQA from 28.36% to 57.46%.
Our results establish a new paradigm for test-time scaling in diffusion-based
LLMs (dLLMs), revealing that the sequence in which masking is performed plays a
critical role in determining performance during inference.
68e4b83be4e093a7044e4e18
Jihoon Lee
68e4b83be4e093a7044e4e19
Hoyeon Moon
68e4b83be4e093a7044e4e1a
Kevin Zhai
claimed_verified
2026-02-20T08:38:57.094Z
65a6c40e3bb0e70b41805319
/avatars/785f90b41f112e93e6470dd397da7c67.svg
Kevin Zhai
k-zhai
user
k-zhai
68e4b83be4e093a7044e4e1b
Arun Kumar Chithanar
claimed_verified
2026-03-09T11:10:31.312Z
64bb94571d40292dd3bae3d1
/avatars/2483a34484a1b1814112ac96020320e4.svg
Arun Kumar Chithanar
akumch
user
akumch
68e4b83be4e093a7044e4e1c
Anit Kumar Sahu
68e4b83be4e093a7044e4e1d
Soummya Kar
68e4b83be4e093a7044e4e1e
Chul Lee
68e4b83be4e093a7044e4e1f
Souradip Chakraborty
68e4b83be4e093a7044e4e20
Amrit Singh Bedi
claimed_verified
2025-10-07T12:26:08.097Z
68b905f152d028b4e78244a0
/avatars/e3b3304529b7d5c6362fb1324f92e285.svg
Amrit Singh Bedi
amrit0714
user
amrit0714
Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive
  Experts
Diffusion-based large language models (dLLMs) are trained flexibly to model
extreme dependence in the data distribution; however, how to best utilize this
information at inference time remains an open problem. In this work, we uncover
an interesting property of these models: dLLMs trained on textual data
implicitly learn a mixture of semi-autoregressive experts, where different
generation orders reveal different specialized behaviors. We show that
committing to any single, fixed inference time schedule, a common practice,
collapses performance by failing to leverage this latent ensemble. To address
this, we introduce HEX (Hidden semiautoregressive EXperts for test-time
scaling), a training-free inference method that ensembles across heterogeneous
block schedules. By doing a majority vote over diverse block-sized generation
paths, HEX robustly avoids failure modes associated with any single fixed
schedule. On reasoning benchmarks such as GSM8K, it boosts accuracy by up to
3.56X (from 24.72% to 88.10%), outperforming top-K margin inference and
specialized fine-tuned methods like GRPO, without additional training. HEX even
yields significant gains on MATH benchmark from 16.40% to 40.00%, scientific
reasoning on ARC-C from 54.18% to 87.80%, and TruthfulQA from 28.36% to 57.46%.
Our results establish a new paradigm for test-time scaling in diffusion-based
LLMs (dLLMs), revealing that the sequence in which masking is performed plays a
critical role in determining performance during inference.
HEX, a training-free inference method for diffusion-based large language models, ensembles diverse generation paths to improve accuracy across various reasoning benchmarks without additional training.
diffusion-based large language models dLLMs semi-autoregressive experts inference time schedule HEX Hidden semiautoregressive EXperts test-time scaling GSM8K MATH ARC-C TruthfulQA