Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model
Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.
6972d788fb12c92b735b7397
Chenghao Fan
claimed_verified
2026-01-23T09:38:32.109Z
641aa5e391e3376a057bbd4c
/avatars/5818797f27444fde078b503774ee081c.svg
Chenghao Fan
Facico
user
Facico
6972d788fb12c92b735b7398
Wen Heng
6972d788fb12c92b735b7399
Bo Li
6972d788fb12c92b735b739a
Sichen Liu
claimed_verified
2026-02-03T10:09:15.814Z
641dbda28dc52733fa4419cf
https://cdn-avatars.huggingface.co/v1/production/uploads/641dbda28dc52733fa4419cf/vdIsa6UlMIaqHinGrYDb-.png
Sichen Liu
Seas0
user
Seas0
6972d788fb12c92b735b739b
Yuxuan Song
6972d788fb12c92b735b739c
Jing Su
6972d788fb12c92b735b739d
Xiaoye Qu
6972d788fb12c92b735b739e
Kai Shen
6972d788fb12c92b735b739f
Wei Wei
Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model
Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.
Stable-DiffCoder demonstrates superior code modeling performance compared to autoregressive baselines through block diffusion continual pretraining and efficient training mechanisms.
diffusion-based language models autoregressive models block diffusion continual pretraining warmup clipped noise schedule supervised fine-tuning code modeling structured code modeling data augmentation