Video-As-Prompt: Unified Semantic Control for Video Generation
Unified, generalizable semantic control in video generation remains a
critical open challenge. Existing methods either introduce artifacts by
enforcing inappropriate pixel-wise priors from structure-based controls, or
rely on non-generalizable, condition-specific finetuning or task-specific
architectures. We introduce Video-As-Prompt (VAP), a new paradigm that reframes
this problem as in-context generation. VAP leverages a reference video as a
direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via
a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture
prevents catastrophic forgetting and is guided by a temporally biased position
embedding that eliminates spurious mapping priors for robust context retrieval.
To power this approach and catalyze future research, we built VAP-Data, the
largest dataset for semantic-controlled video generation with over 100K paired
videos across 100 semantic conditions. As a single unified model, VAP sets a
new state-of-the-art for open-source methods, achieving a 38.7% user preference
rate that rivals leading condition-specific commercial models. VAP's strong
zero-shot generalization and support for various downstream applications mark a
significant advance toward general-purpose, controllable video generation.
68feb9916cdff8b857f470b6
Yuxuan Bian
claimed_verified
2025-10-27T10:27:54.652Z
650447dd52ca06fef957f05d
/avatars/511c11ac9b3cc7a162bda5e07f6ee0a3.svg
Yuxuan BIAN
BianYx
user
BianYx
68feb9916cdff8b857f470b7
Xin Chen
68feb9916cdff8b857f470b8
Zenan Li
68feb9916cdff8b857f470b9
Tiancheng Zhi
68feb9916cdff8b857f470ba
Shen Sang
claimed_verified
2026-03-26T14:05:59.728Z
637d576c43fec4c21637a713
/avatars/9474afce8f834902e0c79de50c98f4fd.svg
Shen Sang
shensang
user
shensang
68feb9916cdff8b857f470bb
Linjie Luo
68feb9916cdff8b857f470bc
Qiang Xu
Video-As-Prompt: Unified Semantic Control for Video Generation
Unified, generalizable semantic control in video generation remains a
critical open challenge. Existing methods either introduce artifacts by
enforcing inappropriate pixel-wise priors from structure-based controls, or
rely on non-generalizable, condition-specific finetuning or task-specific
architectures. We introduce Video-As-Prompt (VAP), a new paradigm that reframes
this problem as in-context generation. VAP leverages a reference video as a
direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via
a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture
prevents catastrophic forgetting and is guided by a temporally biased position
embedding that eliminates spurious mapping priors for robust context retrieval.
To power this approach and catalyze future research, we built VAP-Data, the
largest dataset for semantic-controlled video generation with over 100K paired
videos across 100 semantic conditions. As a single unified model, VAP sets a
new state-of-the-art for open-source methods, achieving a 38.7% user preference
rate that rivals leading condition-specific commercial models. VAP's strong
zero-shot generalization and support for various downstream applications mark a
significant advance toward general-purpose, controllable video generation.
Video-As-Prompt (VAP) uses a reference video to guide a frozen Video Diffusion Transformer via a Mixture-of-Transformers expert, achieving state-of-the-art results in semantic-controlled video generation with strong zero-shot generalization.
Video-As-Prompt Video Diffusion Transformer DiT Mixture-of-Transformers MoT catastrophic forgetting temporally biased position embedding VAP-Data zero-shot generalization