V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation
Generating music that temporally aligns with video events is challenging for existing text-to-music models, which lack fine-grained temporal control. We introduce V2M-Zero, a zero-pair video-to-music generation approach that outputs time-aligned music for video. Our method is motivated by a key observation: temporal synchronization requires matching when and how much change occurs, not what changes. While musical and visual events differ semantically, they exhibit shared temporal structure that can be captured independently within each modality. We capture this structure through event curves computed from intra-modal similarity using pretrained music and video encoders. By measuring temporal change within each modality independently, these curves provide comparable representations across modalities. This enables a simple training strategy: fine-tune a text-to-music model on music-event curves, then substitute video-event curves at inference without cross-modal training or paired data. Across OES-Pub, MovieGenBench-Music, and AIST++, V2M-Zero achieves substantial gains over paired-data baselines: 5-21% higher audio quality, 13-15% better semantic alignment, 21-52% improved temporal synchronization, and 28% higher beat alignment on dance videos. We find similar results via a large crowd-source subjective listening test. Overall, our results validate that temporal alignment through within-modality features, rather than paired cross-modal supervision, is effective for video-to-music generation. Results are available at https://genjib.github.io/v2m_zero/
69b22922206bad330cd7ff74
Yan-Bo Lin
claimed_verified
2026-03-12T09:07:06.310Z
63a715c38729ce5b5640b80b
/avatars/5dc0a167c5dae7756a77d79cfd6334b2.svg
yb
genjib
user
genjib
69b22922206bad330cd7ff75
Jonah Casebeer
69b22922206bad330cd7ff76
Long Mai
admin_assigned
2026-03-12T10:15:44.459Z
672aa83aa0ed47d977837e15
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0Von222J_cLw7tj_QWH7f.png
Long Mai
longmai
user
longmai
69b22922206bad330cd7ff77
Aniruddha Mahapatra
admin_assigned
2026-03-12T10:15:50.160Z
633bd831d5935998f74c4156
/avatars/feb4976ad10dd678ccad2652acf8a611.svg
Aniruddha Mahapatra
aniruddha26398
user
aniruddha26398
69b22922206bad330cd7ff78
Gedas Bertasius
admin_assigned
2026-03-12T10:16:02.733Z
65d60993c2c6418adc99206a
/avatars/fb1f9d5af7022bddf4cb090cd477108d.svg
Gedas Bertasius
gberta
user
gberta
69b22922206bad330cd7ff79
Nicholas J. Bryan
admin_assigned
2026-03-12T10:16:09.620Z
648119ee7a741c7f33f49f25
/avatars/183d2143ba20e1cc8712c63c055aadd7.svg
Nicholas J. Bryan
Njb
user
Njb
V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation
Generating music that temporally aligns with video events is challenging for existing text-to-music models, which lack fine-grained temporal control. We introduce V2M-Zero, a zero-pair video-to-music generation approach that outputs time-aligned music for video. Our method is motivated by a key observation: temporal synchronization requires matching when and how much change occurs, not what changes. While musical and visual events differ semantically, they exhibit shared temporal structure that can be captured independently within each modality. We capture this structure through event curves computed from intra-modal similarity using pretrained music and video encoders. By measuring temporal change within each modality independently, these curves provide comparable representations across modalities. This enables a simple training strategy: fine-tune a text-to-music model on music-event curves, then substitute video-event curves at inference without cross-modal training or paired data. Across OES-Pub, MovieGenBench-Music, and AIST++, V2M-Zero achieves substantial gains over paired-data baselines: 5-21% higher audio quality, 13-15% better semantic alignment, 21-52% improved temporal synchronization, and 28% higher beat alignment on dance videos. We find similar results via a large crowd-source subjective listening test. Overall, our results validate that temporal alignment through within-modality features, rather than paired cross-modal supervision, is effective for video-to-music generation. Results are available at https://genjib.github.io/v2m_zero/
V2M-Zero enables video-to-music generation with improved temporal alignment by using modality-specific event curves derived from pretrained encoders, achieving superior audio quality and synchronization without paired training data.
video-to-music generation temporal synchronization event curves intra-modal similarity pretrained encoders text-to-music model fine-tune cross-modal training paired data audio quality semantic alignment beat alignment