Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos
We introduce Synthetic Visual Genome 2 (SVG2), a large-scale panoptic video scene graph dataset. SVG2 contains over 636K videos with 6.6M objects, 52.0M attributes, and 6.7M relations, providing an order-of-magnitude increase in scale and diversity over prior spatio-temporal scene graph datasets. To create SVG2, we design a fully automated pipeline that combines multi-scale panoptic segmentation, online-offline trajectory tracking with automatic new-object discovery, per-trajectory semantic parsing, and GPT-5-based spatio-temporal relation inference. Building on this resource, we train TRaSER, a video scene graph generation model. TRaSER augments VLMs with a trajectory-aligned token arrangement mechanism and new modules: an object-trajectory resampler and a temporal-window resampler to convert raw videos and panoptic trajectories into compact spatio-temporal scene graphs in a single forward pass. The temporal-window resampler binds visual tokens to short trajectory segments to preserve local motion and temporal semantics, while the object-trajectory resampler aggregates entire trajectories to maintain global context for objects. On the PVSG, VIPSeg, VidOR and SVG2 test datasets, TRaSER improves relation detection by +15 to 20%, object prediction by +30 to 40% over the strongest open-source baselines and by +13% over GPT-5, and attribute prediction by +15%. When TRaSER's generated scene graphs are sent to a VLM for video question answering, it delivers a +1.5 to 4.6% absolute accuracy gain over using video only or video augmented with Qwen2.5-VL's generated scene graphs, demonstrating the utility of explicit spatio-temporal scene graphs as an intermediate representation.
69a4f245bc2d66e68888f4a0
Ziqi Gao
claimed_verified
2026-03-03T10:42:46.932Z
65fba64c3bd9e8f8b78ba8cd
/avatars/2586171de43ec80bd6107b25a6f42b29.svg
Ziqi Gao
UWGZQ
user
UWGZQ
69a4f245bc2d66e68888f4a1
Jieyu Zhang
69a4f245bc2d66e68888f4a2
Wisdom Oluchi Ikezogwo
69a4f245bc2d66e68888f4a3
Jae Sung Park
69a4f245bc2d66e68888f4a4
Tario G. You
69a4f245bc2d66e68888f4a5
Daniel Ogbu
69a4f245bc2d66e68888f4a6
Chenhao Zheng
69a4f245bc2d66e68888f4a7
Weikai Huang
69a4f245bc2d66e68888f4a8
Yinuo Yang
69a4f245bc2d66e68888f4a9
Winson Han
69a4f245bc2d66e68888f4aa
Quan Kong
69a4f245bc2d66e68888f4ab
Rajat Saini
69a4f245bc2d66e68888f4ac
Ranjay Krishna
Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos
We introduce Synthetic Visual Genome 2 (SVG2), a large-scale panoptic video scene graph dataset. SVG2 contains over 636K videos with 6.6M objects, 52.0M attributes, and 6.7M relations, providing an order-of-magnitude increase in scale and diversity over prior spatio-temporal scene graph datasets. To create SVG2, we design a fully automated pipeline that combines multi-scale panoptic segmentation, online-offline trajectory tracking with automatic new-object discovery, per-trajectory semantic parsing, and GPT-5-based spatio-temporal relation inference. Building on this resource, we train TRaSER, a video scene graph generation model. TRaSER augments VLMs with a trajectory-aligned token arrangement mechanism and new modules: an object-trajectory resampler and a temporal-window resampler to convert raw videos and panoptic trajectories into compact spatio-temporal scene graphs in a single forward pass. The temporal-window resampler binds visual tokens to short trajectory segments to preserve local motion and temporal semantics, while the object-trajectory resampler aggregates entire trajectories to maintain global context for objects. On the PVSG, VIPSeg, VidOR and SVG2 test datasets, TRaSER improves relation detection by +15 to 20%, object prediction by +30 to 40% over the strongest open-source baselines and by +13% over GPT-5, and attribute prediction by +15%. When TRaSER's generated scene graphs are sent to a VLM for video question answering, it delivers a +1.5 to 4.6% absolute accuracy gain over using video only or video augmented with Qwen2.5-VL's generated scene graphs, demonstrating the utility of explicit spatio-temporal scene graphs as an intermediate representation.
A large-scale video scene graph dataset and corresponding model are presented that enhance relation and object prediction while improving video question answering through explicit spatio-temporal representations.
panoptic segmentation trajectory tracking semantic parsing spatio-temporal relation inference video scene graph generation VLMs trajectory-aligned token arrangement object-trajectory resampler temporal-window resampler scene graph