Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.
69a0ef99a13deaa4494488a1
Christian Simon
69a0ef99a13deaa4494488a2
Masato Ishii
69a0ef99a13deaa4494488a3
Wei-Yao Wang
69a0ef99a13deaa4494488a4
Koichi Saito
69a0ef99a13deaa4494488a5
Akio Hayakawa
69a0ef99a13deaa4494488a6
Dongseok Shim
69a0ef99a13deaa4494488a7
Zhi Zhong
69a0ef99a13deaa4494488a8
Shuyang Cui
69a0ef99a13deaa4494488a9
Shusuke Takahashi
69a0ef99a13deaa4494488aa
Takashi Shibuya
claimed_verified
2026-02-27T16:41:40.194Z
6650773ca6acfdd2aba7d486
/avatars/d297886ea60dbff98a043caf825820ed.svg
Takashi Shibuya
TakashiShibuyaSony
user
TakashiShibuyaSony
69a0ef99a13deaa4494488ab
Yuki Mitsufuji
Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.
MMHNet enables long-form audio generation from video by integrating hierarchical methods and non-causal Mamba, achieving superior performance over existing video-to-audio approaches.
multimodal hierarchical networks MMHNet video-to-audio generation long-form audio generation non-causal Mamba hierarchical method