Fast-SAM3D: 3Dfy Anything in Images but Faster
SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the first systematic investigation into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline's inherent multi-level heterogeneity: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present Fast-SAM3D, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) Modality-Aware Step Caching to decouple structural evolution from sensitive layout updates; (2) Joint Spatiotemporal Token Carving to concentrate refinement on high-entropy regions; and (3) Spectral-Aware Token Aggregation to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to 2.67times end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation. Our code is released in https://github.com/wlfeng0509/Fast-SAM3D.
6985b9944ad556f294b7ed86
Weilun Feng
claimed_verified
2026-03-09T10:40:40.985Z
68d63e44ccf464a96ac18bcb
/avatars/01b93c353a7231fa9e053ba9edcb2b06.svg
Weilun Feng
wlfeng
user
wlfeng
6985b9944ad556f294b7ed87
Mingqiang Wu
6985b9944ad556f294b7ed88
Zhiliang Chen
6985b9944ad556f294b7ed89
Chuanguang Yang
6985b9944ad556f294b7ed8a
Haotong Qin
6985b9944ad556f294b7ed8b
Yuqi Li
6985b9944ad556f294b7ed8c
Xiaokun Liu
6985b9944ad556f294b7ed8d
Guoxin Fan
6985b9944ad556f294b7ed8e
Zhulin An
6985b9944ad556f294b7ed8f
Libo Huang
6985b9944ad556f294b7ed90
Yulun Zhang
6985b9944ad556f294b7ed91
Michele Magno
6985b9944ad556f294b7ed92
Yongjun Xu
Fast-SAM3D: 3Dfy Anything in Images but Faster
SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the first systematic investigation into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline's inherent multi-level heterogeneity: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present Fast-SAM3D, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) Modality-Aware Step Caching to decouple structural evolution from sensitive layout updates; (2) Joint Spatiotemporal Token Carving to concentrate refinement on high-entropy regions; and (3) Spectral-Aware Token Aggregation to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to 2.67times end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation. Our code is released in https://github.com/wlfeng0509/Fast-SAM3D.
Fast-SAM3D addresses slow inference in 3D reconstruction by dynamically adapting computation to varying complexity through heterogeneity-aware mechanisms that improve efficiency without sacrificing quality.
heterogeneity modality-aware step caching joint spatiotemporal token carving spectral-aware token aggregation end-to-end speedup single-view 3D generation