Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models
Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal large language models into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.
69c686ff8508d7e37629b49a
Wenyue Chen
admin_assigned
2026-03-30T10:17:06.672Z
66c4773aea476bea05fd4d95
/avatars/59547221df76fbabba385ac28f06ff86.svg
wenyue Chen
xishushu
user
xishushu
69c686ff8508d7e37629b49b
Wenjue Chen
69c686ff8508d7e37629b49c
Peng Li
69c686ff8508d7e37629b49d
Qinghe Wang
69c686ff8508d7e37629b49e
Xu Jia
69c686ff8508d7e37629b49f
Heliang Zheng
69c686ff8508d7e37629b4a0
Rongfei Jia
69c686ff8508d7e37629b4a1
Yuan Liu
69c686ff8508d7e37629b4a2
Ronggang Wang
Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models
Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal large language models into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.
Know3D integrates multimodal large language models with 3D generation through latent hidden-state injection, enabling language-controlled back-view synthesis by bridging semantic understanding and geometric reconstruction.
VLM-diffusion-based model multimodal large language models latent hidden-state injection semantic understanding diffusion model 3D generation geometric reconstruction back-view hallucination semantically controllable process