Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video
Foley Control is a lightweight approach to video-guided Foley that keeps
pretrained single-modality models frozen and learns only a small
cross-attention bridge between them. We connect V-JEPA2 video embeddings to a
frozen Stable Audio Open DiT text-to-audio (T2A) model by inserting compact
video cross-attention after the model's existing text cross-attention, so
prompts set global semantics while video refines timing and local dynamics. The
frozen backbones retain strong marginals (video; audio given text) and the
bridge learns the audio-video dependency needed for synchronization -- without
retraining the audio prior. To cut memory and stabilize training, we pool video
tokens before conditioning. On curated video-audio benchmarks, Foley Control
delivers competitive temporal and semantic alignment with far fewer trainable
parameters than recent multi-modal systems, while preserving prompt-driven
controllability and production-friendly modularity (swap/upgrade encoders or
the T2A backbone without end-to-end retraining). Although we focus on
Video-to-Foley, the same bridge design can potentially extend to other audio
modalities (e.g., speech).
68ff50f8f3205b9e39c5f3fb
Ciara Rowles
claimed_verified
2025-10-27T14:16:23.529Z
63357214eb6132ca653020e7
https://cdn-avatars.huggingface.co/v1/production/uploads/63357214eb6132ca653020e7/A_imoyFDx30wgi0guG_s8.png
Ciara
CiaraRowles
user
CiaraRowles
68ff50f8f3205b9e39c5f3fc
Varun Jampani
68ff50f8f3205b9e39c5f3fd
Simon Donné
68ff50f8f3205b9e39c5f3fe
Shimon Vainer
68ff50f8f3205b9e39c5f3ff
Julian Parker
68ff50f8f3205b9e39c5f400
Zach Evans
Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video
Foley Control is a lightweight approach to video-guided Foley that keeps
pretrained single-modality models frozen and learns only a small
cross-attention bridge between them. We connect V-JEPA2 video embeddings to a
frozen Stable Audio Open DiT text-to-audio (T2A) model by inserting compact
video cross-attention after the model's existing text cross-attention, so
prompts set global semantics while video refines timing and local dynamics. The
frozen backbones retain strong marginals (video; audio given text) and the
bridge learns the audio-video dependency needed for synchronization -- without
retraining the audio prior. To cut memory and stabilize training, we pool video
tokens before conditioning. On curated video-audio benchmarks, Foley Control
delivers competitive temporal and semantic alignment with far fewer trainable
parameters than recent multi-modal systems, while preserving prompt-driven
controllability and production-friendly modularity (swap/upgrade encoders or
the T2A backbone without end-to-end retraining). Although we focus on
Video-to-Foley, the same bridge design can potentially extend to other audio
modalities (e.g., speech).
Foley Control uses a small cross-attention bridge to synchronize video and audio without retraining, achieving competitive alignment with fewer parameters and maintaining modularity.
cross-attention V-JEPA2 Stable Audio Open DiT text-to-audio video embeddings audio-video dependency video tokens temporal alignment semantic alignment prompt-driven controllability production-friendly modularity