IGGT: Instance-Grounded Geometry Transformer for Semantic 3D
  Reconstruction
Humans naturally perceive the geometric structure and semantic content of a
3D world as intertwined dimensions, enabling coherent and accurate
understanding of complex scenes. However, most prior approaches prioritize
training large geometry models for low-level 3D reconstruction and treat
high-level spatial understanding in isolation, overlooking the crucial
interplay between these two fundamental aspects of 3D-scene analysis, thereby
limiting generalization and leading to poor performance in downstream 3D
understanding tasks. Recent attempts have mitigated this issue by simply
aligning 3D models with specific language models, thus restricting perception
to the aligned model's capacity and limiting adaptability to downstream tasks.
In this paper, we propose InstanceGrounded Geometry Transformer (IGGT), an
end-to-end large unified transformer to unify the knowledge for both spatial
reconstruction and instance-level contextual understanding. Specifically, we
design a 3D-Consistent Contrastive Learning strategy that guides IGGT to encode
a unified representation with geometric structures and instance-grounded
clustering through only 2D visual inputs. This representation supports
consistent lifting of 2D visual inputs into a coherent 3D scene with explicitly
distinct object instances. To facilitate this task, we further construct
InsScene-15K, a large-scale dataset with high-quality RGB images, poses, depth
maps, and 3D-consistent instance-level mask annotations with a novel data
curation pipeline.
69003b1c22d452aac6dd43a3
Hao Li
claimed_verified
2025-10-28T15:36:13.580Z
667b8de7a68bf81afe668afe
/avatars/aeff10805ff858332e6f6a58735dbbd9.svg
leoli
lifuguan
user
lifuguan
69003b1c22d452aac6dd43a4
Zhengyu Zou
69003b1c22d452aac6dd43a5
Fangfu Liu
69003b1c22d452aac6dd43a6
Xuanyang Zhang
69003b1c22d452aac6dd43a7
Fangzhou Hong
69003b1c22d452aac6dd43a8
Yukang Cao
69003b1c22d452aac6dd43a9
Yushi Lan
69003b1c22d452aac6dd43aa
Manyuan Zhang
69003b1c22d452aac6dd43ab
Gang Yu
69003b1c22d452aac6dd43ac
Dingwen Zhang
69003b1c22d452aac6dd43ad
Ziwei Liu
IGGT: Instance-Grounded Geometry Transformer for Semantic 3D
  Reconstruction
Humans naturally perceive the geometric structure and semantic content of a
3D world as intertwined dimensions, enabling coherent and accurate
understanding of complex scenes. However, most prior approaches prioritize
training large geometry models for low-level 3D reconstruction and treat
high-level spatial understanding in isolation, overlooking the crucial
interplay between these two fundamental aspects of 3D-scene analysis, thereby
limiting generalization and leading to poor performance in downstream 3D
understanding tasks. Recent attempts have mitigated this issue by simply
aligning 3D models with specific language models, thus restricting perception
to the aligned model's capacity and limiting adaptability to downstream tasks.
In this paper, we propose InstanceGrounded Geometry Transformer (IGGT), an
end-to-end large unified transformer to unify the knowledge for both spatial
reconstruction and instance-level contextual understanding. Specifically, we
design a 3D-Consistent Contrastive Learning strategy that guides IGGT to encode
a unified representation with geometric structures and instance-grounded
clustering through only 2D visual inputs. This representation supports
consistent lifting of 2D visual inputs into a coherent 3D scene with explicitly
distinct object instances. To facilitate this task, we further construct
InsScene-15K, a large-scale dataset with high-quality RGB images, poses, depth
maps, and 3D-consistent instance-level mask annotations with a novel data
curation pipeline.
InstanceGrounded Geometry Transformer (IGGT) unifies 3D reconstruction and instance-level understanding using a unified transformer and 3D-Consistent Contrastive Learning, supported by a new dataset InsScene-15K.
InstanceGrounded Geometry Transformer IGGT 3D-Consistent Contrastive Learning 3D reconstruction instance-level contextual understanding 3D scene object instances InsScene-15K