SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing
Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and datasets will be released.
69bcaf2e081b9ad49af57209
Xinyao Zhang
69bcaf2e081b9ad49af5720a
Wenkai Dong
69bcaf2e081b9ad49af5720b
Yuxin Song
69bcaf2e081b9ad49af5720c
Bo Fang
69bcaf2e081b9ad49af5720d
Qi Zhang
69bcaf2e081b9ad49af5720e
Jing Wang
69bcaf2e081b9ad49af5720f
Fan Chen
69bcaf2e081b9ad49af57210
Hui Zhang
69bcaf2e081b9ad49af57211
Haocheng Feng
69bcaf2e081b9ad49af57212
Yu Lu
69bcaf2e081b9ad49af57213
Hang Zhou
69bcaf2e081b9ad49af57214
Chun Yuan
69bcaf2e081b9ad49af57215
Jingdong Wang
SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing
Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and datasets will be released.
SAMA presents a factorized approach to video editing that separates semantic anchoring from motion modeling, enabling instruction-guided edits with preserved motion through pre-trained motion restoration tasks and sparse anchor frame prediction.
semantic anchoring motion alignment video editing semantic tokens video latents sparse anchor frames motion-centric video restoration cube inpainting speed perturbation tube shuffle factorized pre-training supervised fine-tuning zero-shot video editing