End-to-End Video Character Replacement without Structural Guidance
Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha
696705cec5e371f6b235d0f4
Zhengbo Xu
696705cec5e371f6b235d0f5
Jie Ma
claimed_verified
2026-01-14T09:51:30.285Z
641ab8b0a5f876fe30c1ff87
/avatars/a95544619ef05814d58e2b2b8581f223.svg
jiema
unrealMJ
user
unrealMJ
696705cec5e371f6b235d0f6
Ziheng Wang
696705cec5e371f6b235d0f7
Zhan Peng
696705cec5e371f6b235d0f8
Jun Liang
claimed_verified
2026-01-14T09:51:28.253Z
6455afeabda0fbba412d4922
/avatars/367731ce1c71d1e19ff415a52ae4067d.svg
Jun Liang
utopiar
user
utopiar
696705cec5e371f6b235d0f9
Jing Li
End-to-End Video Character Replacement without Structural Guidance
Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha
MoCha enables controllable video character replacement using a single frame mask through condition-aware RoPE and a comprehensive data construction pipeline with specialized datasets.
controllable video character replacement reconstruction-based paradigm per-frame segmentation masks structural guidance facial identity condition-aware RoPE RL-based post-training data construction pipeline high-fidelity rendered dataset expression-driven dataset augmented dataset