Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning
In this report, we introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model. It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation -- even outperforming several larger models. We begin by critically reexamining a widely adopted distillation paradigm in the community: SFT on teacher-generated responses, also known as sequence-level distillation. Although a series of recent works following this scheme have demonstrated remarkable efficiency and strong empirical performance, they are primarily grounded in the SFT perspective. Consequently, these approaches focus predominantly on designing heuristic rules for SFT data filtering, while largely overlooking the core principle of distillation itself -- enabling the student model to learn the teacher's full output distribution so as to inherit its generalization capability. Specifically, we identify three critical limitations in current practice: i) Inadequate representation of the teacher's sequence-level distribution; ii) Misalignment between the teacher's output distribution and the student's learning capacity; and iii) Exposure bias arising from teacher-forced training versus autoregressive inference. In summary, these shortcomings reflect a systemic absence of explicit teacher-student interaction throughout the distillation process, leaving the essence of distillation underexploited. To address these issues, we propose several methodological innovations that collectively form an enhanced sequence-level distillation training pipeline. Remarkably, DASD-4B-Thinking obtains competitive results using only 448K training samples -- an order of magnitude fewer than those employed by most existing open-source efforts. To support community research, we publicly release our models and the training dataset.
69688bbc0ac10a06522f6aeb
Shaotian Yan
claimed_verified
2026-01-15T09:33:27.639Z
6463345cd2044cd1d7c613a8
/avatars/242cbf2479877e836f931d17a6190660.svg
Shaotian
ystluffy
user
ystluffy
69688bbc0ac10a06522f6aec
Kaiyuan Liu
69688bbc0ac10a06522f6aed
Chen Shen
claimed_verified
2026-01-15T09:33:19.260Z
64b73e3830a0b8ff60145a29
/avatars/297469812b57b2ddf7d52b9391d80bde.svg
Chen Shen
zjushenchen
user
zjushenchen
69688bbc0ac10a06522f6aee
Bing Wang
claimed_verified
2026-01-15T09:33:17.091Z
6225b0d87f5fba1007d62fae
https://cdn-avatars.huggingface.co/v1/production/uploads/6225b0d87f5fba1007d62fae/j-QeMTPu7Ob5MKSCEjKKV.jpeg
Bing Wang
wangbing1416
user
wangbing1416
69688bbc0ac10a06522f6aef
Sinan Fan
claimed_verified
2026-01-15T09:33:14.748Z
694a226980a37f293a4ce7c0
/avatars/7a48f4eeb80a1b5688cbfb10a59765a0.svg
Sinan Fan
sinan25
user
sinan25
69688bbc0ac10a06522f6af0
Jun Zhang
69688bbc0ac10a06522f6af1
Yue Wu
69688bbc0ac10a06522f6af2
Zheng Wang
69688bbc0ac10a06522f6af3
Jieping Ye
Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning
In this report, we introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model. It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation -- even outperforming several larger models. We begin by critically reexamining a widely adopted distillation paradigm in the community: SFT on teacher-generated responses, also known as sequence-level distillation. Although a series of recent works following this scheme have demonstrated remarkable efficiency and strong empirical performance, they are primarily grounded in the SFT perspective. Consequently, these approaches focus predominantly on designing heuristic rules for SFT data filtering, while largely overlooking the core principle of distillation itself -- enabling the student model to learn the teacher's full output distribution so as to inherit its generalization capability. Specifically, we identify three critical limitations in current practice: i) Inadequate representation of the teacher's sequence-level distribution; ii) Misalignment between the teacher's output distribution and the student's learning capacity; and iii) Exposure bias arising from teacher-forced training versus autoregressive inference. In summary, these shortcomings reflect a systemic absence of explicit teacher-student interaction throughout the distillation process, leaving the essence of distillation underexploited. To address these issues, we propose several methodological innovations that collectively form an enhanced sequence-level distillation training pipeline. Remarkably, DASD-4B-Thinking obtains competitive results using only 448K training samples -- an order of magnitude fewer than those employed by most existing open-source efforts. To support community research, we publicly release our models and the training dataset.
A lightweight open-source reasoning model achieves state-of-the-art performance through enhanced sequence-level distillation that addresses limitations in current teacher-student knowledge transfer methods.
sequence-level distillation teacher-student distillation SFT heuristic rules output distribution generalization capability exposure bias teacher-forced training autoregressive inference