T3D: Few-Step Diffusion Language Models via Trajectory Self-Distillation with Direct Discriminative Optimization
Diffusion large language models (DLLMs) have the potential to enable fast text generation by decoding multiple tokens in parallel. However, in practice, their inference efficiency is constrained by the need for many refinement steps, while aggressively reducing the number of steps leads to a substantial degradation in generation quality. To alleviate this, we propose a trajectory self-distillation framework that improves few-step decoding by distilling the model's own generative trajectories. We incorporate Direct Discriminative Optimization (DDO), a reverse-KL objective that promotes mode-seeking distillation and encourages the student to concentrate on high-probability teacher modes. Across benchmarks, our approach consistently outperforms strong few-step baselines and standard training under tight step budgets. Although full-step decoding remains superior, we substantially narrow the gap, establishing a strong foundation towards practical few-step DLLMs. The source code is available at https://github.com/Tyrion58/T3D.
698ebb22cace060ff123aeb2
Tunyu Zhang
claimed_verified
2026-02-13T09:36:14.849Z
66b7a545c5a90c1aa84ca807
https://cdn-avatars.huggingface.co/v1/production/uploads/66b7a545c5a90c1aa84ca807/_aJOGL5k-8-75cI9mo7Iy.png
Tunyu Zhang
Tyrion279
user
Tyrion279
698ebb22cace060ff123aeb3
Xinxi Zhang
claimed_verified
2026-02-17T15:51:15.652Z
6410adfda8d20679261c5e44
https://cdn-avatars.huggingface.co/v1/production/uploads/1678814698406-noauth.jpeg
Xinxi Zhang
Xinxi-Zhang
user
Xinxi-Zhang
698ebb22cace060ff123aeb4
Ligong Han
698ebb22cace060ff123aeb5
Haizhou Shi
698ebb22cace060ff123aeb6
Xiaoxiao He
698ebb22cace060ff123aeb7
Zhuowei Li
698ebb22cace060ff123aeb8
Hao Wang
698ebb22cace060ff123aeb9
Kai Xu
698ebb22cace060ff123aeba
Akash Srivastava
698ebb22cace060ff123aebb
Hao Wang
698ebb22cace060ff123aebc
Vladimir Pavlovic
698ebb22cace060ff123aebd
Dimitris N. Metaxas
T3D: Few-Step Diffusion Language Models via Trajectory Self-Distillation with Direct Discriminative Optimization
Diffusion large language models (DLLMs) have the potential to enable fast text generation by decoding multiple tokens in parallel. However, in practice, their inference efficiency is constrained by the need for many refinement steps, while aggressively reducing the number of steps leads to a substantial degradation in generation quality. To alleviate this, we propose a trajectory self-distillation framework that improves few-step decoding by distilling the model's own generative trajectories. We incorporate Direct Discriminative Optimization (DDO), a reverse-KL objective that promotes mode-seeking distillation and encourages the student to concentrate on high-probability teacher modes. Across benchmarks, our approach consistently outperforms strong few-step baselines and standard training under tight step budgets. Although full-step decoding remains superior, we substantially narrow the gap, establishing a strong foundation towards practical few-step DLLMs. The source code is available at https://github.com/Tyrion58/T3D.
A trajectory self-distillation framework with direct discriminative optimization improves few-step decoding efficiency in diffusion large language models while maintaining generation quality.
diffusion large language models trajectory self-distillation self-distillation Direct Discriminative Optimization reverse-KL objective mode-seeking distillation generative trajectories few-step decoding text generation