FastVMT: Eliminating Redundancy in Video Motion Transfer
Video motion transfer aims to synthesize videos by generating visual content according to a text prompt while transferring the motion pattern observed in a reference video. Recent methods predominantly use the Diffusion Transformer (DiT) architecture. To achieve satisfactory runtime, several methods attempt to accelerate the computations in the DiT, but fail to address structural sources of inefficiency. In this work, we identify and remove two types of computational redundancy in earlier work: motion redundancy arises because the generic DiT architecture does not reflect the fact that frame-to-frame motion is small and smooth; gradient redundancy occurs if one ignores that gradients change slowly along the diffusion trajectory. To mitigate motion redundancy, we mask the corresponding attention layers to a local neighborhood such that interaction weights are not computed unnecessarily distant image regions. To exploit gradient redundancy, we design an optimization scheme that reuses gradients from previous diffusion steps and skips unwarranted gradient computations. On average, FastVMT achieves a 3.43x speedup without degrading the visual fidelity or the temporal consistency of the generated videos.
69855dcf4ad556f294b7eb3f
Yue Ma
69855dcf4ad556f294b7eb40
Zhikai Wang
69855dcf4ad556f294b7eb41
Tianhao Ren
69855dcf4ad556f294b7eb42
Mingzhe Zheng
claimed_verified
2026-03-24T09:23:25.588Z
62f0c4abe2999b231e5a893c
/avatars/90da268b877a7ffe6665075c84018a83.svg
Mingzhe Zheng
Dunge0nMaster
user
Dunge0nMaster
69855dcf4ad556f294b7eb43
Hongyu Liu
69855dcf4ad556f294b7eb44
Jiayi Guo
69855dcf4ad556f294b7eb45
Mark Fong
69855dcf4ad556f294b7eb46
Yuxuan Xue
69855dcf4ad556f294b7eb47
Zixiang Zhao
69855dcf4ad556f294b7eb48
Konrad Schindler
69855dcf4ad556f294b7eb49
Qifeng Chen
69855dcf4ad556f294b7eb4a
Linfeng Zhang
FastVMT: Eliminating Redundancy in Video Motion Transfer
Video motion transfer aims to synthesize videos by generating visual content according to a text prompt while transferring the motion pattern observed in a reference video. Recent methods predominantly use the Diffusion Transformer (DiT) architecture. To achieve satisfactory runtime, several methods attempt to accelerate the computations in the DiT, but fail to address structural sources of inefficiency. In this work, we identify and remove two types of computational redundancy in earlier work: motion redundancy arises because the generic DiT architecture does not reflect the fact that frame-to-frame motion is small and smooth; gradient redundancy occurs if one ignores that gradients change slowly along the diffusion trajectory. To mitigate motion redundancy, we mask the corresponding attention layers to a local neighborhood such that interaction weights are not computed unnecessarily distant image regions. To exploit gradient redundancy, we design an optimization scheme that reuses gradients from previous diffusion steps and skips unwarranted gradient computations. On average, FastVMT achieves a 3.43x speedup without degrading the visual fidelity or the temporal consistency of the generated videos.
FastVMT accelerates video motion transfer by addressing computational redundancies in Diffusion Transformer architecture through localized attention masking and gradient reuse optimization.
Diffusion Transformer video motion transfer computational redundancy attention layers gradient reuse diffusion trajectory temporal consistency