RobotArena infty: Scalable Robot Benchmarking via Real-to-Sim
  Translation
The pursuit of robot generalists - instructable agents capable of performing
diverse tasks across diverse environments - demands rigorous and scalable
evaluation. Yet real-world testing of robot policies remains fundamentally
constrained: it is labor-intensive, slow, unsafe at scale, and difficult to
reproduce. Existing simulation benchmarks are similarly limited, as they train
and test policies within the same synthetic domains and cannot assess models
trained from real-world demonstrations or alternative simulation environments.
As policies expand in scope and complexity, these barriers only intensify,
since defining "success" in robotics often hinges on nuanced human judgments of
execution quality. In this paper, we introduce a new benchmarking framework
that overcomes these challenges by shifting VLA evaluation into large-scale
simulated environments augmented with online human feedback. Leveraging
advances in vision-language models, 2D-to-3D generative modeling, and
differentiable rendering, our approach automatically converts video
demonstrations from widely used robot datasets into simulated counterparts.
Within these digital twins, we assess VLA policies using both automated
VLM-guided scoring and scalable human preference judgments collected from
crowdworkers, transforming human involvement from tedious scene setup,
resetting, and safety supervision into lightweight preference comparisons. To
measure robustness, we systematically perturb simulated environments along
multiple axes, such as textures and object placements, stress-testing policy
generalization under controlled variation. The result is a continuously
evolving, reproducible, and scalable benchmark for real-world trained robot
manipulation policies, addressing a critical missing capability in today's
robotics landscape.
6900399b22d452aac6dd4384
Yash Jangir
6900399b22d452aac6dd4385
Yidi Zhang
6900399b22d452aac6dd4386
Kashu Yamazaki
6900399b22d452aac6dd4387
Chenyu Zhang
6900399b22d452aac6dd4388
Kuan-Hsun Tu
claimed_verified
2026-03-23T11:03:52.627Z
69510b8af089e912cc615907
/avatars/c250ec4ba5df8078ad4576d18539a66a.svg
TU
KUAN-HSUN
user
KUAN-HSUN
6900399b22d452aac6dd4389
Tsung-Wei Ke
6900399b22d452aac6dd438a
Lei Ke
claimed_verified
2025-12-12T09:18:34.929Z
6258a6455ea3a0a9b6de3f22
/avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
Lei Ke
lkeab
user
lkeab
6900399b22d452aac6dd438b
Yonatan Bisk
6900399b22d452aac6dd438c
Katerina Fragkiadaki
RobotArena infty: Scalable Robot Benchmarking via Real-to-Sim
  Translation
The pursuit of robot generalists - instructable agents capable of performing
diverse tasks across diverse environments - demands rigorous and scalable
evaluation. Yet real-world testing of robot policies remains fundamentally
constrained: it is labor-intensive, slow, unsafe at scale, and difficult to
reproduce. Existing simulation benchmarks are similarly limited, as they train
and test policies within the same synthetic domains and cannot assess models
trained from real-world demonstrations or alternative simulation environments.
As policies expand in scope and complexity, these barriers only intensify,
since defining "success" in robotics often hinges on nuanced human judgments of
execution quality. In this paper, we introduce a new benchmarking framework
that overcomes these challenges by shifting VLA evaluation into large-scale
simulated environments augmented with online human feedback. Leveraging
advances in vision-language models, 2D-to-3D generative modeling, and
differentiable rendering, our approach automatically converts video
demonstrations from widely used robot datasets into simulated counterparts.
Within these digital twins, we assess VLA policies using both automated
VLM-guided scoring and scalable human preference judgments collected from
crowdworkers, transforming human involvement from tedious scene setup,
resetting, and safety supervision into lightweight preference comparisons. To
measure robustness, we systematically perturb simulated environments along
multiple axes, such as textures and object placements, stress-testing policy
generalization under controlled variation. The result is a continuously
evolving, reproducible, and scalable benchmark for real-world trained robot
manipulation policies, addressing a critical missing capability in today's
robotics landscape.
A new benchmarking framework uses large-scale simulated environments with human feedback to evaluate robot policies, addressing limitations in real-world testing and existing simulation benchmarks.
robot generalists instructable agents rigorous evaluation real-world testing simulation benchmarks VLA evaluation vision-language models 2D-to-3D generative modeling differentiable rendering video demonstrations digital twins automated VLM-guided scoring human preference judgments crowdworkers policy generalization robustness simulated environments