Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models
Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training.
69bb8938ba2b62a5fb0b0d60
Linghao Zhang
69bb8938ba2b62a5fb0b0d61
Jungang Li
claimed_verified
2026-03-23T10:56:14.786Z
64b76528fdb702b3d8641514
https://cdn-avatars.huggingface.co/v1/production/uploads/64b76528fdb702b3d8641514/Ho-uWcQCAEIURM1lhWEWJ.jpeg
Jungang Li
Jungang
user
Jungang
69bb8938ba2b62a5fb0b0d62
Yonghua Hei
69bb8938ba2b62a5fb0b0d63
Sicheng Tao
69bb8938ba2b62a5fb0b0d64
Song Dai
69bb8938ba2b62a5fb0b0d65
Yibo Yan
claimed_verified
2026-03-23T10:56:16.984Z
668def32af57d6e4b5a6d703
/avatars/508bd9eb2e9af1af0febc934f438f33f.svg
yiboyan
Buluchacha
user
Buluchacha
69bb8938ba2b62a5fb0b0d66
Zihao Dongfang
claimed_verified
2026-03-19T12:56:51.383Z
65db5f578c1745678f0ed708
/avatars/4e2de6f5f3a936447b7e391cb14c5346.svg
DONGFANG ZIHAO
UUUserna
user
UUUserna
69bb8938ba2b62a5fb0b0d67
Weiting Liu
69bb8938ba2b62a5fb0b0d68
Chenxi Qin
69bb8938ba2b62a5fb0b0d69
Hanqian Li
claimed_verified
2026-04-13T19:26:21.694Z
660256a2c5665728f7ca0579
/avatars/14b6b0e9b15838a86eeb11ac3161226b.svg
Hanqian Li
Lihanqian
user
Lihanqian
69bb8938ba2b62a5fb0b0d6a
Xin Zou
69bb8938ba2b62a5fb0b0d6b
Jiahao Zhang
69bb8938ba2b62a5fb0b0d6c
Shuhang Xun
69bb8938ba2b62a5fb0b0d6d
Haiyun Jiang
69bb8938ba2b62a5fb0b0d6e
Xuming Hu
Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models
Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training.
Video-supervised fine-tuning in multimodal large language models consistently enhances video performance while often degrading static image benchmarks, with frame sampling frequency determining the extent of this trade-off.
multimodal large language models video-based supervised fine-tuning visual understanding video performance static image benchmarks temporal budget frame sampling instruction-aware Hybrid-Frame strategy