Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence
The pursuit of spatial intelligence fundamentally relies on access to large-scale, fine-grained 3D data. However, existing approaches predominantly construct spatial understanding benchmarks by generating question-answer (QA) pairs from a limited number of manually annotated datasets, rather than systematically annotating new large-scale 3D scenes from raw web data. As a result, their scalability is severely constrained, and model performance is further hindered by domain gaps inherent in these narrowly curated datasets.
  In this work, we propose Holi-Spatial, the first fully automated, large-scale, spatially-aware multimodal dataset, constructed from raw video inputs without human intervention, using the proposed data curation pipeline. Holi-Spatial supports multi-level spatial supervision, ranging from geometrically accurate 3D Gaussian Splatting (3DGS) reconstructions with rendered depth maps to object-level and relational semantic annotations, together with corresponding spatial Question-Answer (QA) pairs.
  Following a principled and systematic pipeline, we further construct Holi-Spatial-4M, the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs spanning diverse geometric, relational, and semantic reasoning tasks.
  Holi-Spatial demonstrates exceptional performance in data curation quality, significantly outperforming existing feed-forward and per-scene optimized methods on datasets such as ScanNet, ScanNet++, and DL3DV. Furthermore, fine-tuning Vision-Language Models (VLMs) on spatial reasoning tasks using this dataset has also led to substantial improvements in model performance.
69af86d0b3ab31dee1281c5d
Yuanyuan Gao
69af86d0b3ab31dee1281c5e
Hao Li
69af86d0b3ab31dee1281c5f
Yifei Liu
69af86d0b3ab31dee1281c60
Xinhao Ji
claimed_verified
2026-03-10T09:45:06.322Z
68633ab108c0b63ef2ec5679
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IEz7UJFvIk_K4DTTRCU_h.png
Xinhao Ji
xinhaoji61
user
xinhaoji61
69af86d0b3ab31dee1281c61
Yuning Gong
69af86d0b3ab31dee1281c62
Yuanjun Liao
69af86d0b3ab31dee1281c63
Fangfu Liu
69af86d0b3ab31dee1281c64
Manyuan Zhang
69af86d0b3ab31dee1281c65
Yuchen Yang
69af86d0b3ab31dee1281c66
Dan Xu
69af86d0b3ab31dee1281c67
Xue Yang
69af86d0b3ab31dee1281c68
Huaxi Huang
69af86d0b3ab31dee1281c69
Hongjie Zhang
69af86d0b3ab31dee1281c6a
Ziwei Liu
69af86d0b3ab31dee1281c6b
Xiao Sun
69af86d0b3ab31dee1281c6c
Dingwen Zhang
69af86d0b3ab31dee1281c6d
Zhihang Zhong
claimed_verified
2026-03-10T18:02:28.346Z
6938f4de790b5cd0f6df6462
/avatars/4f22f0499d96bb749af7e8dba2b0b533.svg
Zhihang Zhong
Zuica96
user
Zuica96
Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence
The pursuit of spatial intelligence fundamentally relies on access to large-scale, fine-grained 3D data. However, existing approaches predominantly construct spatial understanding benchmarks by generating question-answer (QA) pairs from a limited number of manually annotated datasets, rather than systematically annotating new large-scale 3D scenes from raw web data. As a result, their scalability is severely constrained, and model performance is further hindered by domain gaps inherent in these narrowly curated datasets.
  In this work, we propose Holi-Spatial, the first fully automated, large-scale, spatially-aware multimodal dataset, constructed from raw video inputs without human intervention, using the proposed data curation pipeline. Holi-Spatial supports multi-level spatial supervision, ranging from geometrically accurate 3D Gaussian Splatting (3DGS) reconstructions with rendered depth maps to object-level and relational semantic annotations, together with corresponding spatial Question-Answer (QA) pairs.
  Following a principled and systematic pipeline, we further construct Holi-Spatial-4M, the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs spanning diverse geometric, relational, and semantic reasoning tasks.
  Holi-Spatial demonstrates exceptional performance in data curation quality, significantly outperforming existing feed-forward and per-scene optimized methods on datasets such as ScanNet, ScanNet++, and DL3DV. Furthermore, fine-tuning Vision-Language Models (VLMs) on spatial reasoning tasks using this dataset has also led to substantial improvements in model performance.
Holi-Spatial presents the first fully automated, large-scale, spatially-aware multimodal dataset constructed from raw video inputs, supporting multi-level spatial supervision for 3D scene understanding and spatial reasoning tasks.
3D Gaussian Splatting 3DGS Vision-Language Models VLMs spatial reasoning multimodal dataset data curation pipeline geometric accuracy relational semantics semantic annotations 3D reconstruction 3D grounding spatial Question-Answer pairs