TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code will be made publicly available at https://github.com/yaolinli/TimeChat-Captioner.
698ab9841b2dc6b37d61b092
Linli Yao
698ab9841b2dc6b37d61b093
Yuancheng Wei
698ab9841b2dc6b37d61b094
Yaojie Zhang
698ab9841b2dc6b37d61b095
Lei Li
claimed_verified
2026-04-08T08:07:11.157Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
tobiaslee
user
tobiaslee
698ab9841b2dc6b37d61b096
Xinlong Chen
698ab9841b2dc6b37d61b097
Feifan Song
698ab9841b2dc6b37d61b098
Ziyue Wang
698ab9841b2dc6b37d61b099
Kun Ouyang
698ab9841b2dc6b37d61b09a
Yuanxin Liu
claimed_verified
2026-04-08T16:34:49.479Z
6489761dcaea79f577897f98
/avatars/8f56dc9c08dc2b672555602d68509a03.svg
Yuanxin Liu
lyx97
user
lyx97
698ab9841b2dc6b37d61b09b
Lingpeng Kong
698ab9841b2dc6b37d61b09c
Qi Liu
698ab9841b2dc6b37d61b09d
Pengfei Wan
698ab9841b2dc6b37d61b09e
Kun Gai
698ab9841b2dc6b37d61b09f
Yuanxing Zhang
698ab9841b2dc6b37d61b0a0
Xu Sun
TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code will be made publicly available at https://github.com/yaolinli/TimeChat-Captioner.
Omni Dense Captioning introduces a six-dimensional structural schema for generating time-aware audio-visual narratives with explicit timestamps, along with a unified evaluation metric and strong baseline model.
Omni Dense Captioning structured schema script-like captions TimeAware SFT GRPO task-specific rewards dense descriptions audio-visual reasoning temporal grounding