ShapeR: Robust Conditional 3D Shape Generation from Casual Captures
Recent advances in 3D shape generation have achieved impressive results, but most existing methods rely on clean, unoccluded, and well-segmented inputs. Such conditions are rarely met in real-world scenarios. We present ShapeR, a novel approach for conditional 3D object shape generation from casually captured sequences. Given an image sequence, we leverage off-the-shelf visual-inertial SLAM, 3D detection algorithms, and vision-language models to extract, for each object, a set of sparse SLAM points, posed multi-view images, and machine-generated captions. A rectified flow transformer trained to effectively condition on these modalities then generates high-fidelity metric 3D shapes. To ensure robustness to the challenges of casually captured data, we employ a range of techniques including on-the-fly compositional augmentations, a curriculum training scheme spanning object- and scene-level datasets, and strategies to handle background clutter. Additionally, we introduce a new evaluation benchmark comprising 178 in-the-wild objects across 7 real-world scenes with geometry annotations. Experiments show that ShapeR significantly outperforms existing approaches in this challenging setting, achieving an improvement of 2.7x in Chamfer distance compared to state of the art.
696e08453f1837bfb8970b21
Yawar Siddiqui
claimed_verified
2026-01-19T17:36:01.202Z
63fa41a24380ab0cb953655a
/avatars/7b5744208e49b8e4dda0b095d214597e.svg
Yawar Nihal
yawarnihal
user
yawarnihal
696e08453f1837bfb8970b22
Duncan Frost
admin_assigned
2026-01-19T14:44:26.492Z
65fc12c6fc9132a2dfe9e8b3
/avatars/3d04d79721f4ec6b0b090e0bbbf85f70.svg
Duncan Frost
frosd01
user
frosd01
696e08453f1837bfb8970b23
Samir Aroudj
admin_assigned
2026-01-19T14:44:32.397Z
66e9580650afdab5fd6ce00a
https://cdn-avatars.huggingface.co/v1/production/uploads/66e9580650afdab5fd6ce00a/L0sxbhGxl5rrRa9UjLAkK.jpeg
Samir Aroudj
SamirAroudj
user
SamirAroudj
696e08453f1837bfb8970b24
Armen Avetisyan
696e08453f1837bfb8970b25
Henry Howard-Jenkins
admin_assigned
2026-01-19T14:44:41.099Z
65d4991111930466954340ae
/avatars/ffc07be80323aa4b7dadf8849b70ffde.svg
Henry Howard-Jenkins
henryhj
user
henryhj
696e08453f1837bfb8970b26
Daniel DeTone
admin_assigned
2026-01-19T14:44:47.110Z
66f1e84ea4d3e85a77bdb545
/avatars/4e24b0bdd7a8c86ba5f43f4b042817ea.svg
Daniel DeTone
ddetone
user
ddetone
696e08453f1837bfb8970b27
Pierre Moulon
admin_assigned
2026-01-19T14:44:53.004Z
649b0f28c4afb9f5bcf20a27
/avatars/cedf08d01f90494b82eba49b84746f99.svg
Pierre Moulon
TheFrenchLeaf
user
TheFrenchLeaf
696e08453f1837bfb8970b28
Qirui Wu
admin_assigned
2026-01-19T14:44:58.602Z
631db802f318ed8dfd3293fd
/avatars/698cb057456f7868fd3b06acbe08f095.svg
Qirui Wu
qiruiw
user
qiruiw
696e08453f1837bfb8970b29
Zhengqin Li
admin_assigned
2026-01-19T14:45:08.258Z
64ff5b2569219ce3e47893f1
/avatars/36bfb3de85488c0efa75d741eac9c40e.svg
Li
Zhengqin
user
Zhengqin
696e08453f1837bfb8970b2a
Julian Straub
696e08453f1837bfb8970b2b
Richard Newcombe
696e08453f1837bfb8970b2c
Jakob Engel
ShapeR: Robust Conditional 3D Shape Generation from Casual Captures
Recent advances in 3D shape generation have achieved impressive results, but most existing methods rely on clean, unoccluded, and well-segmented inputs. Such conditions are rarely met in real-world scenarios. We present ShapeR, a novel approach for conditional 3D object shape generation from casually captured sequences. Given an image sequence, we leverage off-the-shelf visual-inertial SLAM, 3D detection algorithms, and vision-language models to extract, for each object, a set of sparse SLAM points, posed multi-view images, and machine-generated captions. A rectified flow transformer trained to effectively condition on these modalities then generates high-fidelity metric 3D shapes. To ensure robustness to the challenges of casually captured data, we employ a range of techniques including on-the-fly compositional augmentations, a curriculum training scheme spanning object- and scene-level datasets, and strategies to handle background clutter. Additionally, we introduce a new evaluation benchmark comprising 178 in-the-wild objects across 7 real-world scenes with geometry annotations. Experiments show that ShapeR significantly outperforms existing approaches in this challenging setting, achieving an improvement of 2.7x in Chamfer distance compared to state of the art.
ShapeR generates high-fidelity 3D shapes from casual image sequences using visual-inertial SLAM, 3D detection, and vision-language models with rectified flow transformer conditioning.
3D shape generation visual-inertial SLAM 3D detection algorithms vision-language models rectified flow transformer Chamfer distance