RobotArena $\infty$: Scalable Robot Benchmarking via Real-to-Sim Translation

TL;DR

RobotArena ∞ tackles the scalability bottleneck in robot policy evaluation by translating real-world demonstration videos into large, simulated environments and scoring executions with vision-language models and crowdsourced preferences. The framework fuses VLM-based scene understanding, 2D-to-3D asset generation, differentiable rendering, and automated calibrations to produce physics-enabled digital twins seeded from diverse datasets. It evaluates multiple generalist robot policies across hundreds of environments and 7k+ human preferences, uncovering cross-dataset generalization gaps and robustness weaknesses while revealing consistent model rankings. By releasing the benchmark environments and evaluation code, RobotArena ∞ aims to provide a reproducible, extensible platform that accelerates progress toward robust, generalist robotic policies.

Abstract

The pursuit of robot generalists - instructable agents capable of performing diverse tasks across diverse environments - demands rigorous and scalable evaluation. Yet real-world testing of robot policies remains fundamentally constrained: it is labor-intensive, slow, unsafe at scale, and difficult to reproduce. Existing simulation benchmarks are similarly limited, as they train and test policies within the same synthetic domains and cannot assess models trained from real-world demonstrations or alternative simulation environments. As policies expand in scope and complexity, these barriers only intensify, since defining "success" in robotics often hinges on nuanced human judgments of execution quality. In this paper, we introduce a new benchmarking framework that overcomes these challenges by shifting VLA evaluation into large-scale simulated environments augmented with online human feedback. Leveraging advances in vision-language models, 2D-to-3D generative modeling, and differentiable rendering, our approach automatically converts video demonstrations from widely used robot datasets into simulated counterparts. Within these digital twins, we assess VLA policies using both automated VLM-guided scoring and scalable human preference judgments collected from crowdworkers, transforming human involvement from tedious scene setup, resetting, and safety supervision into lightweight preference comparisons. To measure robustness, we systematically perturb simulated environments along multiple axes, such as textures and object placements, stress-testing policy generalization under controlled variation. The result is a continuously evolving, reproducible, and scalable benchmark for real-world trained robot manipulation policies, addressing a critical missing capability in today's robotics landscape.

Figures (20)

• Figure 1: RobotArena $\infty$ provides a scalable and extensible robot benchmarking framework by automating environment construction and evaluation. It automatically generates simulated environment seeded from real videos, deploys robot policies, and evaluates them using VLMs and crowdsourced workers that cast preferences between pairs of execution videos. The simulated environments are derived from both in-distribution and out-of-distribution videos, enabling rigorous tests of generalization in contemporary VLAs.
• Figure 2: Automated video-to-simulation translation in RobotArena $\infty$. Given a frame from a robot demonstration video, we automatically create a corresponding simulated environment.
• Figure 3: Automated robot-camera calibration through differentiable rendering of pose-conditioned 3D robot Gaussians.
• Figure 4: We obtain task progress scores for execution videos automatically by prompting Gemini with a shuffled frame sequence and the language instruction, following ma2024visionlanguagemodelsincontext.
• Figure 5: We show simulation environments in RobotArena $\infty$ seeded from videos demonstrations in the datasets of Bridge, RH20T and DROID.