WaDi: Weight Direction-aware Distillation for One-step Image Synthesis
Despite the impressive performance of diffusion models such as Stable Diffusion (SD) in image generation, their slow inference limits practical deployment. Recent works accelerate inference by distilling multi-step diffusion into one-step generators. To better understand the distillation mechanism, we analyze U-Net/DiT weight changes between one-step students and their multi-step teacher counterparts. Our analysis reveals that changes in weight direction significantly exceed those in weight norm, highlighting it as the key factor during distillation. Motivated by this insight, we propose the Low-rank Rotation of weight Direction (LoRaD), a parameter-efficient adapter tailored to one-step diffusion distillation. LoRaD is designed to model these structured directional changes using learnable low-rank rotation matrices. We further integrate LoRaD into Variational Score Distillation (VSD), resulting in Weight Direction-aware Distillation (WaDi)-a novel one-step distillation framework. WaDi achieves state-of-the-art FID scores on COCO 2014 and COCO 2017 while using only approximately 10% of the trainable parameters of the U-Net/DiT. Furthermore, the distilled one-step model demonstrates strong versatility and scalability, generalizing well to various downstream tasks such as controllable generation, relation inversion, and high-resolution synthesis.
69b4074653b5fe8c5e117f59
Lei Wang
claimed_verified
2026-04-17T06:59:31.802Z
660a2f348b022f13fd6bb15b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ONDILmYtX8JPCAJRuqsAW.png
Lei Wang
gudaochangsheng
user
gudaochangsheng
69b4074653b5fe8c5e117f5a
Yang Cheng
69b4074653b5fe8c5e117f5b
Senmao Li
69b4074653b5fe8c5e117f5c
Ge Wu
69b4074653b5fe8c5e117f5d
Yaxing Wang
69b4074653b5fe8c5e117f5e
Jian Yang
WaDi: Weight Direction-aware Distillation for One-step Image Synthesis
Despite the impressive performance of diffusion models such as Stable Diffusion (SD) in image generation, their slow inference limits practical deployment. Recent works accelerate inference by distilling multi-step diffusion into one-step generators. To better understand the distillation mechanism, we analyze U-Net/DiT weight changes between one-step students and their multi-step teacher counterparts. Our analysis reveals that changes in weight direction significantly exceed those in weight norm, highlighting it as the key factor during distillation. Motivated by this insight, we propose the Low-rank Rotation of weight Direction (LoRaD), a parameter-efficient adapter tailored to one-step diffusion distillation. LoRaD is designed to model these structured directional changes using learnable low-rank rotation matrices. We further integrate LoRaD into Variational Score Distillation (VSD), resulting in Weight Direction-aware Distillation (WaDi)-a novel one-step distillation framework. WaDi achieves state-of-the-art FID scores on COCO 2014 and COCO 2017 while using only approximately 10% of the trainable parameters of the U-Net/DiT. Furthermore, the distilled one-step model demonstrates strong versatility and scalability, generalizing well to various downstream tasks such as controllable generation, relation inversion, and high-resolution synthesis.
Diffusion model distillation techniques using low-rank rotation matrices achieve high-quality image generation with reduced parameters and enhanced versatility across downstream tasks.
diffusion models Stable Diffusion U-Net DiT distillation weight direction low-rank rotation matrices parameter-efficient adapter Variational Score Distillation FID scores controllable generation relation inversion high-resolution synthesis