AVControl: Efficient Framework for Training Audio-Visual Controls
Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.
69c6548c2de2ee20094eded2
Matan Ben-Yosef
claimed_verified
2026-03-27T11:32:14.462Z
630389a70907b9a115c1af93
https://cdn-avatars.huggingface.co/v1/production/uploads/630389a70907b9a115c1af93/0iGogiSByXVAvBLHiazoA.jpeg
Matan
matanby
user
matanby
69c6548c2de2ee20094eded3
Tavi Halperin
claimed_verified
2026-03-30T09:10:08.099Z
6369676b85fef3ca96e5e2f1
/avatars/aa33e95ca2d9e70a95623459c2fba9f2.svg
Tavi Halperin
tavihalperin
user
tavihalperin
69c6548c2de2ee20094eded4
Naomi Ken Korem
69c6548c2de2ee20094eded5
Mohammad Salama
69c6548c2de2ee20094eded6
Harel Cain
claimed_verified
2026-03-30T09:11:49.808Z
6315a88999d400946c3d0ae6
/avatars/4a3178215a4114880194dece88c7c7ca.svg
Harel Cain
harelcain
user
harelcain
69c6548c2de2ee20094eded7
Asaf Joseph
69c6548c2de2ee20094eded8
Anthony Chen
69c6548c2de2ee20094eded9
Urska Jelercic
69c6548c2de2ee20094ededa
Ofir Bibi
claimed_verified
2026-04-16T08:19:14.797Z
631f58935ba8c026340b377c
https://cdn-avatars.huggingface.co/v1/production/uploads/631f58935ba8c026340b377c/4yoHLdNE99VBb7ji_Mzzj.jpeg
Ofir Bibi
ofirbibi
user
ofirbibi
AVControl: Efficient Framework for Training Audio-Visual Controls
Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.
AVControl enables efficient, modular audio-visual generation by training control modalities as separate LoRA adapters on a parallel canvas within LTX-2, achieving superior performance on diverse control tasks while requiring minimal computational resources.
LoRA LTX-2 audio-visual foundation model parallel canvas attention layers in-context methods VACE Benchmark depth-guided generation pose-guided generation inpainting outpainting camera control audio-visual benchmarks spatially-aligned controls camera trajectory sparse motion control video editing modular audio-visual controls compute-efficient data-efficient