Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion
  Transformers
Diffusion Transformers (DiTs) deliver state-of-the-art generative performance
but their quadratic training cost with sequence length makes large-scale
pretraining prohibitively expensive. Token dropping can reduce training cost,
yet na\"ive strategies degrade representations, and existing methods are either
parameter-heavy or fail at high drop ratios. We present SPRINT, Sparse--Dense
Residual Fusion for Efficient Diffusion Transformers, a simple method that
enables aggressive token dropping (up to 75%) while preserving quality. SPRINT
leverages the complementary roles of shallow and deep layers: early layers
process all tokens to capture local detail, deeper layers operate on a sparse
subset to cut computation, and their outputs are fused through residual
connections. Training follows a two-stage schedule: long masked pre-training
for efficiency followed by short full-token fine-tuning to close the
train--inference gap. On ImageNet-1K 256x256, SPRINT achieves 9.8x training
savings with comparable FID/FDD, and at inference, its Path-Drop Guidance (PDG)
nearly halves FLOPs while improving quality. These results establish SPRINT as
a simple, effective, and general solution for efficient DiT training.
69013d33646208eac0d1f16b
Dogyun Park
69013d33646208eac0d1f16c
Moayed Haji-Ali
69013d33646208eac0d1f16d
Yanyu Li
69013d33646208eac0d1f16e
Willi Menapace
69013d33646208eac0d1f16f
Sergey Tulyakov
69013d33646208eac0d1f170
Hyunwoo J. Kim
69013d33646208eac0d1f171
Aliaksandr Siarohin
69013d33646208eac0d1f172
Anil Kag
claimed_verified
2026-03-17T08:32:26.239Z
69b84d69b6bcc376261b1b55
/avatars/87692477b406c4fc7f689316b81d50a7.svg
Anil Kag
akag-snap
user
akag-snap
Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion
  Transformers
Diffusion Transformers (DiTs) deliver state-of-the-art generative performance
but their quadratic training cost with sequence length makes large-scale
pretraining prohibitively expensive. Token dropping can reduce training cost,
yet na\"ive strategies degrade representations, and existing methods are either
parameter-heavy or fail at high drop ratios. We present SPRINT, Sparse--Dense
Residual Fusion for Efficient Diffusion Transformers, a simple method that
enables aggressive token dropping (up to 75%) while preserving quality. SPRINT
leverages the complementary roles of shallow and deep layers: early layers
process all tokens to capture local detail, deeper layers operate on a sparse
subset to cut computation, and their outputs are fused through residual
connections. Training follows a two-stage schedule: long masked pre-training
for efficiency followed by short full-token fine-tuning to close the
train--inference gap. On ImageNet-1K 256x256, SPRINT achieves 9.8x training
savings with comparable FID/FDD, and at inference, its Path-Drop Guidance (PDG)
nearly halves FLOPs while improving quality. These results establish SPRINT as
a simple, effective, and general solution for efficient DiT training.
SPRINT, a method for efficient training of Diffusion Transformers, achieves significant training cost reduction and maintains quality through aggressive token dropping and residual fusion.
Diffusion Transformers DiTs token dropping sparse-dense residual fusion local detail residual connections masked pre-training full-token fine-tuning ImageNet-1K FID/FDD Path-Drop Guidance FLOPs