EasyV2V: A High-quality Instruction-based Video Editing Framework
While image editing has advanced rapidly, video editing remains less explored, facing challenges in consistency, control, and generalization. We study the design space of data, architecture, and control, and introduce EasyV2V, a simple and effective framework for instruction-based video editing. On the data side, we compose existing experts with fast inverses to build diverse video pairs, lift image edit pairs into videos via single-frame supervision and pseudo pairs with shared affine motion, mine dense-captioned clips for video pairs, and add transition supervision to teach how edits unfold. On the model side, we observe that pretrained text-to-video models possess editing capability, motivating a simplified design. Simple sequence concatenation for conditioning with light LoRA fine-tuning suffices to train a strong model. For control, we unify spatiotemporal control via a single mask mechanism and support optional reference images. Overall, EasyV2V works with flexible inputs, e.g., video+text, video+mask+text, video+mask+reference+text, and achieves state-of-the-art video editing results, surpassing concurrent and commercial systems. Project page: https://snap-research.github.io/easyv2v/
694562cafbf17e708e1862de
Jinjie Mai
694562cafbf17e708e1862df
Chaoyang Wang
694562cafbf17e708e1862e0
Guocheng Gordon Qian
694562cafbf17e708e1862e1
Willi Menapace
694562cafbf17e708e1862e2
Sergey Tulyakov
694562cafbf17e708e1862e3
Bernard Ghanem
694562cafbf17e708e1862e4
Peter Wonka
694562cafbf17e708e1862e5
Ashkan Mirzaei
EasyV2V: A High-quality Instruction-based Video Editing Framework
While image editing has advanced rapidly, video editing remains less explored, facing challenges in consistency, control, and generalization. We study the design space of data, architecture, and control, and introduce EasyV2V, a simple and effective framework for instruction-based video editing. On the data side, we compose existing experts with fast inverses to build diverse video pairs, lift image edit pairs into videos via single-frame supervision and pseudo pairs with shared affine motion, mine dense-captioned clips for video pairs, and add transition supervision to teach how edits unfold. On the model side, we observe that pretrained text-to-video models possess editing capability, motivating a simplified design. Simple sequence concatenation for conditioning with light LoRA fine-tuning suffices to train a strong model. For control, we unify spatiotemporal control via a single mask mechanism and support optional reference images. Overall, EasyV2V works with flexible inputs, e.g., video+text, video+mask+text, video+mask+reference+text, and achieves state-of-the-art video editing results, surpassing concurrent and commercial systems. Project page: https://snap-research.github.io/easyv2v/
EasyV2V framework enhances video editing by combining diverse data sources, leveraging pretrained text-to-video models with LoRA fine-tuning, and implementing unified spatiotemporal control, achieving top results.
EasyV2V fast inverses single-frame supervision pseudo pairs dense-captioned clips transition supervision pretrained text-to-video models LoRA fine-tuning spatiotemporal control mask mechanism reference images