TV2TV: A Unified Framework for Interleaved Language and Video Generation
Video generation models are rapidly advancing, but can still struggle with complex video outputs that require significant semantic branching or repeated high-level reasoning about what should happen next. In this paper, we introduce a new class of omni video-text models that integrate ideas from recent LM reasoning advances to address this challenge. More specifically, we present TV2TV, a unified generative modeling framework which decomposes video generation into an interleaved text and video generation process. TV2TV jointly learns language modeling (next-token prediction) and video flow matching (next-frame prediction) using a Mixture-of-Transformers (MoT) architecture. At inference time, TV2TV decides when to alternate between generating text and video frames, allowing the model to "think in words" about subsequent content before ``acting in pixels'' to produce frames. This design offloads much of the responsibility for deciding what should happen next to the language modeling tower, enabling improved visual quality and prompt alignment of generated videos. It also enables fine-grained controllability, allowing users to modify the video generation trajectory through text interventions at any point in the process. In controlled experiments on video game data, TV2TV demonstrates substantial improvements in both visual quality and controllability. TV2TV also scales to natural videos, as we show by augmenting sports videos with interleaved natural language action descriptions using vision-language models (VLMs). Training TV2TV on this corpus yields strong visual quality and prompt alignment, showcasing the model's ability to reason about and generate complex real-world action sequences. Together, these results highlight TV2TV as a promising step toward video generation with open-ended textual reasoning and control.
6932468c6d1060ca587a2677
Xiaochuang Han
claimed_verified
2025-12-10T09:10:21.189Z
635ea3969f24f6db0a1e2d0b
https://cdn-avatars.huggingface.co/v1/production/uploads/635ea3969f24f6db0a1e2d0b/RxaJKAx4MYMShb58yuUxy.png
Xiaochuang Han
xhan77
user
xhan77
6932468c6d1060ca587a2678
Youssef Emad
6932468c6d1060ca587a2679
Melissa Hall
6932468c6d1060ca587a267a
John Nguyen
claimed_verified
2025-12-16T15:09:18.262Z
648a14e002c8497f58ebff62
/avatars/0d30f7bd843ac94f317d8cfc53256450.svg
John Nguyen
ngjhn
user
ngjhn
6932468c6d1060ca587a267b
Karthik Padthe
claimed_verified
2025-12-08T08:32:38.641Z
64c01530bd4b58768d8b46b5
/avatars/0f3c0cdab16e3078b37569d73ca03792.svg
Karthik padthe
Karthikpadthe
user
Karthikpadthe
6932468c6d1060ca587a267c
Liam Robbins
6932468c6d1060ca587a267d
Amir Bar
6932468c6d1060ca587a267e
Delong Chen
claimed_verified
2026-01-16T14:38:13.184Z
630491107424d937fa3258be
/avatars/b8bd81bc8544674ee26b78702afdb87c.svg
Delong Chen
chendelong
user
chendelong
6932468c6d1060ca587a267f
Michal Drozdzal
6932468c6d1060ca587a2680
Maha Elbayad
claimed_verified
2025-12-12T09:17:57.553Z
64e243dff8934329dd72098f
https://cdn-avatars.huggingface.co/v1/production/uploads/64e243dff8934329dd72098f/JbB4zvlendHSRXjx_iC0j.jpeg
Maha Elbayad
elbayadm
user
elbayadm
6932468c6d1060ca587a2681
Yushi Hu
6932468c6d1060ca587a2682
Shang-Wen Li
claimed_verified
2025-12-08T08:32:36.529Z
64e3a7b8e12618b261fff364
/avatars/860a5011f150537eb923f76d354f1bc5.svg
Shang-Wen Daniel Li
swdanielli
user
swdanielli
6932468c6d1060ca587a2683
Sreya Dutta Roy
6932468c6d1060ca587a2684
Jakob Verbeek
6932468c6d1060ca587a2685
XuDong Wang
6932468c6d1060ca587a2686
Marjan Ghazvininejad
claimed_verified
2025-12-10T09:10:18.961Z
660f0fd377a1e2509aa5a679
/avatars/e04ef05bed0bf6cefdc7e3e39674e2f9.svg
Marjan Ghazvininejad
mghazvininejad
user
mghazvininejad
6932468c6d1060ca587a2687
Luke Zettlemoyer
6932468c6d1060ca587a2688
Emily Dinan
TV2TV: A Unified Framework for Interleaved Language and Video Generation
Video generation models are rapidly advancing, but can still struggle with complex video outputs that require significant semantic branching or repeated high-level reasoning about what should happen next. In this paper, we introduce a new class of omni video-text models that integrate ideas from recent LM reasoning advances to address this challenge. More specifically, we present TV2TV, a unified generative modeling framework which decomposes video generation into an interleaved text and video generation process. TV2TV jointly learns language modeling (next-token prediction) and video flow matching (next-frame prediction) using a Mixture-of-Transformers (MoT) architecture. At inference time, TV2TV decides when to alternate between generating text and video frames, allowing the model to "think in words" about subsequent content before ``acting in pixels'' to produce frames. This design offloads much of the responsibility for deciding what should happen next to the language modeling tower, enabling improved visual quality and prompt alignment of generated videos. It also enables fine-grained controllability, allowing users to modify the video generation trajectory through text interventions at any point in the process. In controlled experiments on video game data, TV2TV demonstrates substantial improvements in both visual quality and controllability. TV2TV also scales to natural videos, as we show by augmenting sports videos with interleaved natural language action descriptions using vision-language models (VLMs). Training TV2TV on this corpus yields strong visual quality and prompt alignment, showcasing the model's ability to reason about and generate complex real-world action sequences. Together, these results highlight TV2TV as a promising step toward video generation with open-ended textual reasoning and control.
A new video generation model, TV2TV, integrates text and video generation using a Mixture-of-Transformers to improve visual quality and controllability by leveraging language modeling for high-level reasoning.
video generation models omni video-text models TV2TV unified generative modeling framework language modeling video flow matching Mixture-of-Transformers text and video generation process next-token prediction next-frame prediction language modeling tower prompt alignment video game data natural videos vision-language models visual quality controllability textual reasoning