Watch Before You Answer: Learning from Visually Grounded Post-Training
It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.
69d60afcc6ed9968ac5c6563
Yuxuan Zhang
claimed_verified
2026-04-08T11:40:50.469Z
63b908d0e3c78740d8e950d0
/avatars/3e80075e92aebdfea712f70b00d5ec7d.svg
Yuxuan Zhang
Reacherx
user
Reacherx
69d60afcc6ed9968ac5c6564
EunJeong Hwang
69d60afcc6ed9968ac5c6565
Huaisong Zhang
69d60afcc6ed9968ac5c6566
Penghui Du
claimed_verified
2026-04-08T11:40:52.046Z
647076467fd7ecdbd0ea03b1
/avatars/6e090ea5f88977c6f70544175094c2a6.svg
Penghui Du
eternaldolphin
user
eternaldolphin
69d60afcc6ed9968ac5c6567
Yiming Jia
69d60afcc6ed9968ac5c6568
Dongfu Jiang
69d60afcc6ed9968ac5c6569
Xuan He
69d60afcc6ed9968ac5c656a
Shenhui Zhang
69d60afcc6ed9968ac5c656b
Ping Nie
claimed_verified
2026-04-08T16:34:08.889Z
65358802a920f38780b3248a
/avatars/9415510b598079973c2b0436ad12db9c.svg
Ping Nie
pingnieuk
user
pingnieuk
69d60afcc6ed9968ac5c656c
Peter West
69d60afcc6ed9968ac5c656d
Kelsey R. Allen
Watch Before You Answer: Learning from Visually Grounded Post-Training
It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.
Vision-language models face challenges in video understanding due to text-based biases in benchmarks and datasets, which are addressed through VidGround, a method that uses only visually grounded questions for post-training to improve performance.
vision-language models video understanding multimodal modeling post-training visual grounding data curation RL-based post-training algorithms