GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars
Photo-realistic and controllable 3D avatars are crucial for various applications such as virtual and mixed reality (VR/MR), telepresence, gaming, and film production. Traditional methods for avatar creation often involve time-consuming scanning and reconstruction processes for each avatar, which limits their scalability. Furthermore, these methods do not offer the flexibility to sample new identities or modify existing ones. On the other hand, by learning a strong prior from data, generative models provide a promising alternative to traditional reconstruction methods, easing the time constraints for both data capture and processing. Additionally, generative methods enable downstream applications beyond reconstruction, such as editing and stylization. Nonetheless, the research on generative 3D avatars is still in its infancy, and therefore current methods still have limitations such as creating static avatars, lacking photo-realism, having incomplete facial details, or having limited drivability. To address this, we propose a text-conditioned generative model that can generate photo-realistic facial avatars of diverse identities, with more complete details like hair, eyes and mouth interior, and which can be driven through a powerful non-parametric latent expression space. Specifically, we integrate the generative and editing capabilities of latent diffusion models with a strong prior model for avatar expression driving.
  Our model can generate and control high-fidelity avatars, even those out-of-distribution. We also highlight its potential for downstream applications, including avatar editing and single-shot avatar reconstruction.
66ce87b28ec9fda2cf61078d
Keqiang Sun
admin_assigned
2024-08-28T08:16:15.959Z
645ca9f3a19f3e64bbed7534
/avatars/37fde5fd978bc03a3a02c30102557432.svg
Keqiang Sun
kqsun
user
kqsun
66ce87b28ec9fda2cf61078e
Amin Jourabloo
66ce87b28ec9fda2cf61078f
Riddhish Bhalodia
66ce87b28ec9fda2cf610790
Moustafa Meshry
66ce87b28ec9fda2cf610791
Yu Rong
claimed_verified
2025-07-31T21:01:32.065Z
642eecbf9b2484d7d8526781
https://cdn-avatars.huggingface.co/v1/production/uploads/642eecbf9b2484d7d8526781/4IvGbd66s49Wx5pZyZGHA.png
Yu Rong
Swrooy
user
Swrooy
66ce87b28ec9fda2cf610792
Zhengyu Yang
66ce87b28ec9fda2cf610793
Thu Nguyen-Phuoc
admin_assigned
2024-08-28T08:17:42.507Z
6304b663c2f4f2d4929ad200
/avatars/805d4dab512fc53d8bd6bcb60db3821b.svg
Thu Nguyen-Phuoc
TNP23
user
TNP23
66ce87b28ec9fda2cf610794
Christian Haene
66ce87b28ec9fda2cf610795
Jiu Xu
66ce87b28ec9fda2cf610796
Sam Johnson
66ce87b28ec9fda2cf610797
Hongsheng Li
admin_assigned
2024-08-28T08:18:33.911Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
hsli-cuhk
user
hsli-cuhk
66ce87b28ec9fda2cf610798
Sofien Bouaziz
GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars
Photo-realistic and controllable 3D avatars are crucial for various applications such as virtual and mixed reality (VR/MR), telepresence, gaming, and film production. Traditional methods for avatar creation often involve time-consuming scanning and reconstruction processes for each avatar, which limits their scalability. Furthermore, these methods do not offer the flexibility to sample new identities or modify existing ones. On the other hand, by learning a strong prior from data, generative models provide a promising alternative to traditional reconstruction methods, easing the time constraints for both data capture and processing. Additionally, generative methods enable downstream applications beyond reconstruction, such as editing and stylization. Nonetheless, the research on generative 3D avatars is still in its infancy, and therefore current methods still have limitations such as creating static avatars, lacking photo-realism, having incomplete facial details, or having limited drivability. To address this, we propose a text-conditioned generative model that can generate photo-realistic facial avatars of diverse identities, with more complete details like hair, eyes and mouth interior, and which can be driven through a powerful non-parametric latent expression space. Specifically, we integrate the generative and editing capabilities of latent diffusion models with a strong prior model for avatar expression driving.
  Our model can generate and control high-fidelity avatars, even those out-of-distribution. We also highlight its potential for downstream applications, including avatar editing and single-shot avatar reconstruction.
A text-conditioned generative model creates photo-realistic 3D avatars with complete facial details and controllable expressions using latent diffusion models and a prior expression driving model.
latent diffusion models generative models 3D avatars photo-realistic facial details latent expression space avatar editing single-shot reconstruction