Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial
  Representations
Humans learn abstract concepts through multisensory synergy, and once formed,
such representations can often be recalled from a single modality. Inspired by
this principle, we introduce Concerto, a minimalist simulation of human concept
learning for spatial cognition, combining 3D intra-modal self-distillation with
2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more
coherent and informative spatial features, as demonstrated by zero-shot
visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised
models by 14.2% and 4.8%, respectively, as well as their feature concatenation,
in linear probing for 3D scene perception. With full fine-tuning, Concerto sets
new SOTA results across multiple scene understanding benchmarks (e.g., 80.7%
mIoU on ScanNet). We further present a variant of Concerto tailored for
video-lifted point cloud spatial understanding, and a translator that linearly
projects Concerto representations into CLIP's language space, enabling
open-world perception. These results highlight that Concerto emerges spatial
representations with superior fine-grained geometric and semantic consistency.
690034a222d452aac6dd4346
Yujia Zhang
claimed_verified
2026-03-04T09:25:31.765Z
6513eaee4f1682e440809f95
/avatars/858914670c91a30d4c4ce1204d2cab80.svg
yujia
yujia0913
user
yujia0913
690034a222d452aac6dd4347
Xiaoyang Wu
claimed_verified
2025-10-28T15:36:29.001Z
643e5d6a1d0e956d94bb3608
https://cdn-avatars.huggingface.co/v1/production/uploads/643e5d6a1d0e956d94bb3608/mTnu7Dts2RmDklHlp9Gqu.jpeg
Xiaoyang Wu
Gofinge
user
Gofinge
690034a222d452aac6dd4348
Yixing Lao
690034a222d452aac6dd4349
Chengyao Wang
claimed_verified
2025-10-28T15:36:26.782Z
6423e35b30b0e4ab36dd1b16
https://cdn-avatars.huggingface.co/v1/production/uploads/6423e35b30b0e4ab36dd1b16/pea6LVDS9PQAxvQt9GUiZ.jpeg
Wang Chengyao
wcy1122
user
wcy1122
690034a222d452aac6dd434a
Zhuotao Tian
690034a222d452aac6dd434b
Naiyan Wang
690034a222d452aac6dd434c
Hengshuang Zhao
Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial
  Representations
Humans learn abstract concepts through multisensory synergy, and once formed,
such representations can often be recalled from a single modality. Inspired by
this principle, we introduce Concerto, a minimalist simulation of human concept
learning for spatial cognition, combining 3D intra-modal self-distillation with
2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more
coherent and informative spatial features, as demonstrated by zero-shot
visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised
models by 14.2% and 4.8%, respectively, as well as their feature concatenation,
in linear probing for 3D scene perception. With full fine-tuning, Concerto sets
new SOTA results across multiple scene understanding benchmarks (e.g., 80.7%
mIoU on ScanNet). We further present a variant of Concerto tailored for
video-lifted point cloud spatial understanding, and a translator that linearly
projects Concerto representations into CLIP's language space, enabling
open-world perception. These results highlight that Concerto emerges spatial
representations with superior fine-grained geometric and semantic consistency.
Concerto, a minimalist model combining 3D self-distillation and 2D-3D joint embedding, achieves superior spatial feature learning and outperforms existing models in scene understanding and open-world perception.
3D intra-modal self-distillation 2D-3D cross-modal joint embedding zero-shot visualizations linear probing 3D scene perception ScanNet mIoU video-lifted point cloud CLIP's language space open-world perception fine-grained geometric and semantic consistency