HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds
We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.
69e19353860e47828575b3bd
Team HY-World
69e19353860e47828575b3be
Chenjie Cao
69e19353860e47828575b3bf
Xuhui Zuo
69e19353860e47828575b3c0
Zhenwei Wang
69e19353860e47828575b3c1
Yisu Zhang
69e19353860e47828575b3c2
Junta Wu
69e19353860e47828575b3c3
Zhenyang Liu
69e19353860e47828575b3c4
Yuning Gong
69e19353860e47828575b3c5
Yang Liu
69e19353860e47828575b3c6
Bo Yuan
69e19353860e47828575b3c7
Chao Zhang
69e19353860e47828575b3c8
Coopers Li
69e19353860e47828575b3c9
Dongyuan Guo
69e19353860e47828575b3ca
Fan Yang
69e19353860e47828575b3cb
Haiyu Zhang
69e19353860e47828575b3cc
Hang Cao
69e19353860e47828575b3cd
Jianchen Zhu
69e19353860e47828575b3ce
Jiaxin Lin
69e19353860e47828575b3cf
Jie Xiao
69e19353860e47828575b3d0
Jihong Zhang
69e19353860e47828575b3d1
Junlin Yu
69e19353860e47828575b3d2
Lei Wang
69e19353860e47828575b3d3
Lifu Wang
69e19353860e47828575b3d4
Lilin Wang
69e19353860e47828575b3d5
Linus
69e19353860e47828575b3d6
Minghui Chen
69e19353860e47828575b3d7
Peng He
69e19353860e47828575b3d8
Penghao Zhao
69e19353860e47828575b3d9
Qi Chen
69e19353860e47828575b3da
Rui Chen
69e19353860e47828575b3db
Rui Shao
69e19353860e47828575b3dc
Sicong Liu
69e19353860e47828575b3dd
Wangchen Qin
69e19353860e47828575b3de
Xiaochuan Niu
69e19353860e47828575b3df
Xiang Yuan
69e19353860e47828575b3e0
Yi Sun
69e19353860e47828575b3e1
Yifei Tang
69e19353860e47828575b3e2
Yifu Sun
69e19353860e47828575b3e3
Yihang Lian
69e19353860e47828575b3e4
Yonghao Tan
69e19353860e47828575b3e5
Yuhong Liu
69e19353860e47828575b3e6
Yuyang Yin
69e19353860e47828575b3e7
Zhiyuan Min
69e19353860e47828575b3e8
Tengfei Wang
69e19353860e47828575b3e9
Chunchao Guo
HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds
We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.
HY-World 2.0 is a multi-modal world model framework that generates high-fidelity 3D Gaussian Splatting scenes from diverse inputs using specialized modules for panorama generation, trajectory planning, world expansion, and composition, along with an enhanced rendering platform for interactive 3D exploration.
multi-modal world model 3D Gaussian Splatting HY-Pano 2.0 WorldNav WorldStereo 2.0 WorldMirror 2.0 keyframe-based view generation feed-forward model 3D world representations interactive exploration rendering platform