FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation
Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct.
6980487a6676f933227066aa
Siyang He
6980487a6676f933227066ab
Qiqi Wang
6980487a6676f933227066ac
Xiaoran Liu
claimed_verified
2026-02-02T16:52:14.672Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Xiaoran Liu (SII)
SII-xrliu
user
SII-xrliu
6980487a6676f933227066ad
Hongnan Ma
6980487a6676f933227066ae
Yiwei Shi
6980487a6676f933227066af
Yuerong Song
6980487a6676f933227066b0
Ying Zhu
claimed_verified
2026-02-10T09:27:45.279Z
69158ffc0153b85a677dcc46
/avatars/c9c5f60522f2a8f370d790ea9938b090.svg
Ying Zhu
Auraithm
user
Auraithm
6980487a6676f933227066b1
Tianyi Liang
6980487a6676f933227066b2
Zengfeng Huang
6980487a6676f933227066b3
Ziwei He
6980487a6676f933227066b4
Xipeng Qiu
FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion Language Models via Frequency-Guided Generation
Despite the non-autoregressive potential of diffusion language models (dLLMs), existing decoding strategies demonstrate positional bias, failing to fully unlock the potential of arbitrary generation. In this work, we delve into the inherent spectral characteristics of dLLMs and present the first frequency-domain analysis showing that low-frequency components in hidden states primarily encode global structural information and long-range dependencies, while high-frequency components are responsible for characterizing local details. Based on this observation, we propose FourierSampler, which leverages a frequency-domain sliding window mechanism to dynamically guide the model to achieve a "structure-to-detail" generation. FourierSampler outperforms other inference enhancement strategies on LLADA and SDAR, achieving relative improvements of 20.4% on LLaDA1.5-8B and 16.0% on LLaDA-8B-Instruct. It notably surpasses similarly sized autoregressive models like Llama3.1-8B-Instruct.
Frequency-domain analysis of diffusion language models reveals that low-frequency components encode global structure while high-frequency components capture local details, enabling improved generation through FourierSampler's dynamic frequency-domain sliding window mechanism.
diffusion language models spectral characteristics frequency-domain analysis hidden states low-frequency components high-frequency components FourierSampler frequency-domain sliding window mechanism arbitrary generation positional bias