Evaluating Gemini Robotics Policies in a Veo World Simulator
Generative world models hold significant potential for simulating interactions with visuomotor policies in varied environments. Frontier video models can enable generation of realistic observations and environment interactions in a scalable and general manner. However, the use of video models in robotics has been limited primarily to in-distribution evaluations, i.e., scenarios that are similar to ones used to train the policy or fine-tune the base video model. In this report, we demonstrate that video models can be used for the entire spectrum of policy evaluation use cases in robotics: from assessing nominal performance to out-of-distribution (OOD) generalization, and probing physical and semantic safety. We introduce a generative evaluation system built upon a frontier video foundation model (Veo). The system is optimized to support robot action conditioning and multi-view consistency, while integrating generative image-editing and multi-view completion to synthesize realistic variations of real-world scenes along multiple axes of generalization. We demonstrate that the system preserves the base capabilities of the video model to enable accurate simulation of scenes that have been edited to include novel interaction objects, novel visual backgrounds, and novel distractor objects. This fidelity enables accurately predicting the relative performance of different policies in both nominal and OOD conditions, determining the relative impact of different axes of generalization on policy performance, and performing red teaming of policies to expose behaviors that violate physical or semantic safety constraints. We validate these capabilities through 1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator.
693b83939874a2a5e4ffb2a4
Gemini Robotics Team
693b83939874a2a5e4ffb2a5
Coline Devin
693b83939874a2a5e4ffb2a6
Yilun Du
admin_assigned
2025-12-12T13:14:15.439Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
yilundu
user
yilundu
693b83939874a2a5e4ffb2a7
Debidatta Dwibedi
693b83939874a2a5e4ffb2a8
Ruiqi Gao
693b83939874a2a5e4ffb2a9
Abhishek Jindal
693b83939874a2a5e4ffb2aa
Thomas Kipf
admin_assigned
2025-12-12T13:14:33.451Z
6413572b6cd62eb3ba2024e9
https://cdn-avatars.huggingface.co/v1/production/uploads/6413572b6cd62eb3ba2024e9/UCtMdeV8_N6w-SBnTDXBy.jpeg
Thomas Kipf
tkipf
user
tkipf
693b83939874a2a5e4ffb2ab
Sean Kirmani
admin_assigned
2025-12-12T13:14:39.448Z
648c85f33ffc11989bddcce1
/avatars/bdde73b95a36e32cb2975656cca46022.svg
Sean Kirmani
skirmani
user
skirmani
693b83939874a2a5e4ffb2ac
Fangchen Liu
admin_assigned
2025-12-12T13:14:44.842Z
63609ef63605bd411c1c7509
/avatars/3fadc3c4117cdf17d25f3efc545388cd.svg
Fangchen Liu
fangchenliu
user
fangchenliu
693b83939874a2a5e4ffb2ad
Anirudha Majumdar
693b83939874a2a5e4ffb2ae
Andrew Marmon
admin_assigned
2025-12-12T13:14:54.023Z
60ac2ec408805574fcf076f4
/avatars/a8cd60f39ba89939c5841252dc9d742b.svg
Andrew Marmon
acoadmarmon
user
acoadmarmon
693b83939874a2a5e4ffb2af
Carolina Parada
admin_assigned
2025-12-12T13:14:59.800Z
65b7c3e87817e067a5b66cf1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KysLXVoFmkPM2pQMgDlsb.jpeg
Carolina Parada
CarolinaParada
user
CarolinaParada
693b83939874a2a5e4ffb2b0
Yulia Rubanova
693b83939874a2a5e4ffb2b1
Dhruv Shah
693b83939874a2a5e4ffb2b2
Vikas Sindhwani
693b83939874a2a5e4ffb2b3
Jie Tan
693b83939874a2a5e4ffb2b4
Fei Xia
693b83939874a2a5e4ffb2b5
Ted Xiao
693b83939874a2a5e4ffb2b6
Sherry Yang
693b83939874a2a5e4ffb2b7
Wenhao Yu
693b83939874a2a5e4ffb2b8
Allan Zhou
Evaluating Gemini Robotics Policies in a Veo World Simulator
Generative world models hold significant potential for simulating interactions with visuomotor policies in varied environments. Frontier video models can enable generation of realistic observations and environment interactions in a scalable and general manner. However, the use of video models in robotics has been limited primarily to in-distribution evaluations, i.e., scenarios that are similar to ones used to train the policy or fine-tune the base video model. In this report, we demonstrate that video models can be used for the entire spectrum of policy evaluation use cases in robotics: from assessing nominal performance to out-of-distribution (OOD) generalization, and probing physical and semantic safety. We introduce a generative evaluation system built upon a frontier video foundation model (Veo). The system is optimized to support robot action conditioning and multi-view consistency, while integrating generative image-editing and multi-view completion to synthesize realistic variations of real-world scenes along multiple axes of generalization. We demonstrate that the system preserves the base capabilities of the video model to enable accurate simulation of scenes that have been edited to include novel interaction objects, novel visual backgrounds, and novel distractor objects. This fidelity enables accurately predicting the relative performance of different policies in both nominal and OOD conditions, determining the relative impact of different axes of generalization on policy performance, and performing red teaming of policies to expose behaviors that violate physical or semantic safety constraints. We validate these capabilities through 1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator.
A generative evaluation system using a frontier video model (Veo) enables comprehensive policy evaluation in robotics, including nominal performance, out-of-distribution generalization, and safety checks.
generative world models video models policy evaluation out-of-distribution (OOD) generalization generative evaluation system frontier video foundation model Veo robot action conditioning multi-view consistency generative image-editing multi-view completion bimanual manipulator