Layer-Aware Video Composition via Split-then-Merge
We present Split-then-Merge (StM), a novel framework designed to enhance control in generative video composition and address its data scarcity problem. Unlike conventional methods relying on annotated datasets or handcrafted rules, StM splits a large corpus of unlabeled videos into dynamic foreground and background layers, then self-composes them to learn how dynamic subjects interact with diverse scenes. This process enables the model to learn the complex compositional dynamics required for realistic video generation. StM introduces a novel transformation-aware training pipeline that utilizes a multi-layer fusion and augmentation to achieve affordance-aware composition, alongside an identity-preservation loss that maintains foreground fidelity during blending. Experiments show StM outperforms SoTA methods in both quantitative benchmarks and in humans/VLLM-based qualitative evaluations. More details are available at our project page: https://split-then-merge.github.io
692d8e058d366933b9cd153a
Ozgur Kara
claimed_verified
2025-12-01T16:24:34.705Z
645d1eb67c7258d904e66c96
/avatars/804097caccc89ecc0b7407fef3da920f.svg
Ozgur Kara
ozgurkara
user
ozgurkara
692d8e058d366933b9cd153b
Yujia Chen
692d8e058d366933b9cd153c
Ming-Hsuan Yang
692d8e058d366933b9cd153d
James M. Rehg
692d8e058d366933b9cd153e
Wen-Sheng Chu
692d8e058d366933b9cd153f
Du Tran
Layer-Aware Video Composition via Split-then-Merge
We present Split-then-Merge (StM), a novel framework designed to enhance control in generative video composition and address its data scarcity problem. Unlike conventional methods relying on annotated datasets or handcrafted rules, StM splits a large corpus of unlabeled videos into dynamic foreground and background layers, then self-composes them to learn how dynamic subjects interact with diverse scenes. This process enables the model to learn the complex compositional dynamics required for realistic video generation. StM introduces a novel transformation-aware training pipeline that utilizes a multi-layer fusion and augmentation to achieve affordance-aware composition, alongside an identity-preservation loss that maintains foreground fidelity during blending. Experiments show StM outperforms SoTA methods in both quantitative benchmarks and in humans/VLLM-based qualitative evaluations. More details are available at our project page: https://split-then-merge.github.io
A novel framework, Split-then-Merge, improves generative video composition through unsupervised learning of foreground-background interactions, achieving state-of-the-art performance.
Split-then-Merge generative video composition dynamic foreground background layers self-composition compositional dynamics transformation-aware training multi-layer fusion augmentation affordance-aware composition identity-preservation loss