Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds
We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.
69154dffa1b06ca3cc81351e
Weihao Tan
69154dffa1b06ca3cc81351f
Xiangyang Li
69154dffa1b06ca3cc813520
Yunhao Fang
69154dffa1b06ca3cc813521
Heyuan Yao
69154dffa1b06ca3cc813522
Shi Yan
69154dffa1b06ca3cc813523
Hao Luo
69154dffa1b06ca3cc813524
Tenglong Ao
69154dffa1b06ca3cc813525
Huihui Li
69154dffa1b06ca3cc813526
Hongbin Ren
69154dffa1b06ca3cc813527
Bairen Yi
claimed_verified
2025-11-17T10:31:49.043Z
6369d92f64aad59d4d44d362
/avatars/73956400cfbfd53116aefc17b3c9f0fd.svg
Yi
Bairen
user
Bairen
69154dffa1b06ca3cc813528
Yujia Qin
69154dffa1b06ca3cc813529
Bo An
69154dffa1b06ca3cc81352a
Libin Liu
69154dffa1b06ca3cc81352b
Guang Shi
Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds
We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.
Lumine, a vision-language model-based agent, completes complex missions in real-time across different 3D open-world environments with human-like efficiency and zero-shot cross-game generalization.
vision-language model end-to-end 3D open-world environments human-like interaction real-time zero-shot cross-game generalization