REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation

TL;DR

REALM introduces a high-fidelity real-to-sim benchmark for evaluating generalization in Vision-Language-Action robotic manipulation, pairing a large perturbation space with aligned control and real-world validation. It demonstrates that sim-to-real correlation is strong enough to serve as a proxy for real-world performance while revealing persistent gaps in semantic and behavioral generalization across state-of-the-art VLA models. The study provides a scalable, extensible benchmark (REALM-base and REALM-articulated) and rigorous evaluation metrics, highlighting that robustness under perturbations remains a key challenge for current VLAs. These findings support the use of realistic simulators to systematically diagnose weaknesses and guide future improvements in generalization, control alignment, and embodiment diversity.

Abstract

Vision-Language-Action (VLA) models empower robots to understand and execute tasks described by natural language instructions. However, a key challenge lies in their ability to generalize beyond the specific environments and conditions they were trained on, which is presently difficult and expensive to evaluate in the real-world. To address this gap, we present REALM, a new simulation environment and benchmark designed to evaluate the generalization capabilities of VLA models, with a specific emphasis on establishing a strong correlation between simulated and real-world performance through high-fidelity visuals and aligned robot control. Our environment offers a suite of 15 perturbation factors, 7 manipulation skills, and more than 3,500 objects. Finally, we establish two task sets that form our benchmark and evaluate the π_{0}, π_{0}-FAST, and GR00T N1.5 VLA models, showing that generalization and robustness remain an open challenge. More broadly, we also show that simulation gives us a valuable proxy for the real-world and allows us to systematically probe for and quantify the weaknesses and failure modes of VLAs. Project page: https://martin-sedlacek.com/realm

Figures (10)

• Figure 1: REALM is a large-scale real-to-sim aligned simulation environment and benchmark for generalization in robotic manipulation. It supports 7 distinct manipulation skills (top left) and stress-tests them against 15 perturbations (bottom left). Through empirical validation, we show that evaluation results in simulation are strongly correlated to real-world performance.
• Figure 2: Visualization of tasks from REALM-base and REALM-articulated. The full benchmark consists of 8 base tasks (6 shown) and 2 articulated tasks (both shown), which are used for the large-scale evaluation in Section \ref{['section:Results']}.
• Figure 3: Visualization of 9 out of 15 perturbations from three categories. Visual perturbations alter the observations in pixel space, but do not require the policy to adjust its behavior or understand a different phrasing of an instruction. Semantic perturbations require understanding different aspects of human language that can be present in robot commands. Behavioral perturbations require changes to robot movement when solving a task compared to the default setting. Some perturbations can encompass multiple categories - e.g. changing the manipulated object (bottom middle) is both visual and behavioral.
• Figure 4: Visualization of the aligned control. We show a trajectory replay in simulation using the default controller (left) and our aligned control (right). Yellow trajectory represents the ground truth from a real robot and blue is from simulation. Our system identification results in significantly more realistic trajectory following.
• Figure 5: Simulation of our set-up. To measure the real-to-sim gap, we compare the task progression on a real (unseen) set-up in our lab (left) and its digital cousin in sim (right).