PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual
  Observations
Modeling and rendering photorealistic avatars is of crucial importance in
many applications. Existing methods that build a 3D avatar from visual
observations, however, struggle to reconstruct clothed humans. We introduce
PhysAvatar, a novel framework that combines inverse rendering with inverse
physics to automatically estimate the shape and appearance of a human from
multi-view video data along with the physical parameters of the fabric of their
clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
spatio-temporal mesh tracking as well as a physically based inverse renderer to
estimate the intrinsic material properties. PhysAvatar integrates a physics
simulator to estimate the physical parameters of the garments using
gradient-based optimization in a principled manner. These novel capabilities
enable PhysAvatar to create high-quality novel-view renderings of avatars
dressed in loose-fitting clothes under motions and lighting conditions not seen
in the training data. This marks a significant advancement towards modeling
photorealistic digital humans using physically based inverse rendering with
physics in the loop. Our project website is at:
https://qingqing-zhao.github.io/PhysAvatar
6614ca518cb1d029bac0f88d
Yang Zheng
6614ca518cb1d029bac0f88e
Qingqing Zhao
6614ca518cb1d029bac0f88f
Guandao Yang
claimed_verified
2024-04-25T06:55:08.036Z
64547ca006728ff79a38e7a5
https://cdn-avatars.huggingface.co/v1/production/uploads/64547ca006728ff79a38e7a5/xfTVQXMSfRv-IbljFpswQ.jpeg
Guandao Yang
guandao
user
guandao
6614ca518cb1d029bac0f890
Wang Yifan
6614ca518cb1d029bac0f891
Donglai Xiang
6614ca518cb1d029bac0f892
Florian Dubost
6614ca518cb1d029bac0f893
Dmitry Lagun
6614ca518cb1d029bac0f894
Thabo Beeler
6614ca518cb1d029bac0f895
Federico Tombari
6614ca518cb1d029bac0f896
Leonidas Guibas
6614ca518cb1d029bac0f897
Gordon Wetzstein
PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual
  Observations
Modeling and rendering photorealistic avatars is of crucial importance in
many applications. Existing methods that build a 3D avatar from visual
observations, however, struggle to reconstruct clothed humans. We introduce
PhysAvatar, a novel framework that combines inverse rendering with inverse
physics to automatically estimate the shape and appearance of a human from
multi-view video data along with the physical parameters of the fabric of their
clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
spatio-temporal mesh tracking as well as a physically based inverse renderer to
estimate the intrinsic material properties. PhysAvatar integrates a physics
simulator to estimate the physical parameters of the garments using
gradient-based optimization in a principled manner. These novel capabilities
enable PhysAvatar to create high-quality novel-view renderings of avatars
dressed in loose-fitting clothes under motions and lighting conditions not seen
in the training data. This marks a significant advancement towards modeling
photorealistic digital humans using physically based inverse rendering with
physics in the loop. Our project website is at:
https://qingqing-zhao.github.io/PhysAvatar
PhysAvatar integrates inverse rendering and physics simulation to reconstruct and render realistic avatars in various dynamic and lighting conditions using multi-view video data.
inverse rendering inverse physics mesh-aligned 4D Gaussian spatio-temporal mesh tracking physically based inverse renderer physics simulator gradient-based optimization photorealistic digital humans