CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation
Camera-controllable video generation aims to synthesize videos with flexible and physically plausible camera movements. However, existing methods either provide imprecise camera control from text prompts or rely on labor-intensive manual camera trajectory parameters, limiting their use in automated scenarios. To address these issues, we propose a novel Vision-Language-Camera model, termed CT-1 (Camera Transformer 1), a specialized model designed to transfer spatial reasoning knowledge to video generation by accurately estimating camera trajectories. Built upon vision-language modules and a Diffusion Transformer model, CT-1 employs a Wavelet-based Regularization Loss in the frequency domain to effectively learn complex camera trajectory distributions. These trajectories are integrated into a video diffusion model to enable spatially aware camera control that aligns with user intentions. To facilitate the training of CT-1, we design a dedicated data curation pipeline and construct CT-200K, a large-scale dataset containing over 47M frames. Experimental results demonstrate that our framework successfully bridges the gap between spatial reasoning and video synthesis, yielding faithful and high-quality camera-controllable videos and improving camera control accuracy by 25.7% over prior methods.
69dc4c372b6b8dc66ffeeb45
Haoyu Zhao
69dc4c372b6b8dc66ffeeb46
Zihao Zhang
69dc4c372b6b8dc66ffeeb47
Jiaxi Gu
69dc4c372b6b8dc66ffeeb48
Haoran Chen
69dc4c372b6b8dc66ffeeb49
Qingping Zheng
69dc4c372b6b8dc66ffeeb4a
Pin Tang
69dc4c372b6b8dc66ffeeb4b
Yeyin Jin
69dc4c372b6b8dc66ffeeb4c
Yuang Zhang
69dc4c372b6b8dc66ffeeb4d
Junqi Cheng
69dc4c372b6b8dc66ffeeb4e
Zenghui Lu
69dc4c372b6b8dc66ffeeb4f
Peng Shu
69dc4c372b6b8dc66ffeeb50
Zuxuan Wu
69dc4c372b6b8dc66ffeeb51
Yu-Gang Jiang
CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation
Camera-controllable video generation aims to synthesize videos with flexible and physically plausible camera movements. However, existing methods either provide imprecise camera control from text prompts or rely on labor-intensive manual camera trajectory parameters, limiting their use in automated scenarios. To address these issues, we propose a novel Vision-Language-Camera model, termed CT-1 (Camera Transformer 1), a specialized model designed to transfer spatial reasoning knowledge to video generation by accurately estimating camera trajectories. Built upon vision-language modules and a Diffusion Transformer model, CT-1 employs a Wavelet-based Regularization Loss in the frequency domain to effectively learn complex camera trajectory distributions. These trajectories are integrated into a video diffusion model to enable spatially aware camera control that aligns with user intentions. To facilitate the training of CT-1, we design a dedicated data curation pipeline and construct CT-200K, a large-scale dataset containing over 47M frames. Experimental results demonstrate that our framework successfully bridges the gap between spatial reasoning and video synthesis, yielding faithful and high-quality camera-controllable videos and improving camera control accuracy by 25.7% over prior methods.
A Vision-Language-Camera model called CT-1 generates videos with accurate camera control by learning camera trajectories through diffusion transformers and wavelet regularization loss.
vision-language modules Diffusion Transformer model Wavelet-based Regularization Loss camera trajectory estimation video diffusion model spatial reasoning camera control accuracy