SAM 3D Body: Robust Full-Body Human Mesh Recovery
We introduce SAM 3D Body (3DB), a promptable model for single-image full-body 3D human mesh recovery (HMR) that demonstrates state-of-the-art performance, with strong generalization and consistent accuracy in diverse in-the-wild conditions. 3DB estimates the human pose of the body, feet, and hands. It is the first model to use a new parametric mesh representation, Momentum Human Rig (MHR), which decouples skeletal structure and surface shape. 3DB employs an encoder-decoder architecture and supports auxiliary prompts, including 2D keypoints and masks, enabling user-guided inference similar to the SAM family of models. We derive high-quality annotations from a multi-stage annotation pipeline that uses various combinations of manual keypoint annotation, differentiable optimization, multi-view geometry, and dense keypoint detection. Our data engine efficiently selects and processes data to ensure data diversity, collecting unusual poses and rare imaging conditions. We present a new evaluation dataset organized by pose and appearance categories, enabling nuanced analysis of model behavior. Our experiments demonstrate superior generalization and substantial improvements over prior methods in both qualitative user preference studies and traditional quantitative analysis. Both 3DB and MHR are open-source.
699678401268a6b79e0d0275
Xitong Yang
699678401268a6b79e0d0276
Devansh Kukreja
699678401268a6b79e0d0277
Don Pinkus
699678401268a6b79e0d0278
Anushka Sagar
699678401268a6b79e0d0279
Taosha Fan
699678401268a6b79e0d027a
Jinhyung Park
699678401268a6b79e0d027b
Soyong Shin
699678401268a6b79e0d027c
Jinkun Cao
699678401268a6b79e0d027d
Jiawei Liu
699678401268a6b79e0d027e
Nicolas Ugrinovic
699678401268a6b79e0d027f
Matt Feiszli
699678401268a6b79e0d0280
Jitendra Malik
699678401268a6b79e0d0281
Piotr Dollar
699678401268a6b79e0d0282
Kris Kitani
SAM 3D Body: Robust Full-Body Human Mesh Recovery
We introduce SAM 3D Body (3DB), a promptable model for single-image full-body 3D human mesh recovery (HMR) that demonstrates state-of-the-art performance, with strong generalization and consistent accuracy in diverse in-the-wild conditions. 3DB estimates the human pose of the body, feet, and hands. It is the first model to use a new parametric mesh representation, Momentum Human Rig (MHR), which decouples skeletal structure and surface shape. 3DB employs an encoder-decoder architecture and supports auxiliary prompts, including 2D keypoints and masks, enabling user-guided inference similar to the SAM family of models. We derive high-quality annotations from a multi-stage annotation pipeline that uses various combinations of manual keypoint annotation, differentiable optimization, multi-view geometry, and dense keypoint detection. Our data engine efficiently selects and processes data to ensure data diversity, collecting unusual poses and rare imaging conditions. We present a new evaluation dataset organized by pose and appearance categories, enabling nuanced analysis of model behavior. Our experiments demonstrate superior generalization and substantial improvements over prior methods in both qualitative user preference studies and traditional quantitative analysis. Both 3DB and MHR are open-source.
A promptable 3D human mesh recovery model using a novel parametric representation and encoder-decoder architecture achieves state-of-the-art performance with strong generalization across diverse conditions.
3D human mesh recovery encoder-decoder architecture parametric mesh representation Momentum Human Rig 2D keypoints masks auxiliary prompts differentiable optimization multi-view geometry dense keypoint detection data engine evaluation dataset qualitative user preference studies quantitative analysis