Capturing Visual Environment Structure Correlates with Control Performance

TL;DR

This paper tackles the cost of evaluating generalist robotic policies by proposing a state-regression proxy: learning to decode a unified, ground-truth environment state from images. The state is a concatenation of per-object and global scene attributes, enabling a lightweight, architecture-agnostic regression objective whose accuracy reliably predicts downstream policy performance across multiple simulators and even transfers to real-world tasks. The authors demonstrate that this state-prediction proxy outperforms prior, object-centric proxies (e.g., segmentation, depth) in ranking backbone suitability and policy success, while also offering computational efficiency. Overall, the work provides a practical, principled approach for selecting visual representations and suggests that encoding latent physical state is a promising objective for robust, generalizable control in robotics.

Abstract

The choice of visual representation is key to scaling generalist robot policies. However, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state -- including geometry, object structure, and physical attributes -- from images. Leveraging simulation environments with access to ground-truth state, we show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics and enabling efficient representation selection. More broadly, our study provides insight into the representational properties that support generalizable manipulation, suggesting that learning to encode the latent physical state of the environment is a promising objective for control.

Figures (9)

• Figure 1: Simulation environments provide access to full world state labels for free (left), enabling our proxy task — state prediction from visual inputs. This proxy strongly correlates with downstream policy success across environments and architectures (right, results for SimplerEnv environment).
• Figure 2: Our framework for efficient visual representation selection for control. We capitalize on the availability of ground truth world state information in simulators and propose a universal, compact encoding of the states, together with a light-weight state prediction head (bottom). We demonstrate a strong correlation between our proxy objective and downstream policy performance (top).
• Figure 3: Evaluation of a diverse set of visual backbones in different robotic simulation environments. From top left to bottom right: The visually simple Metaworld (a) favors more traditional ImageNet-pretrained representations. RoboCasa (b) requires precise objet localization and thus favors with strong object priors, whereas realistic SimplerEnv environments (c, d) benefit from real world robot data pre-training.
• Figure 4: Correlation between state prediction score and success rate of the policies. Our proposed proxy task shows a strong correlation and MMRV score in all four different environments.
• Figure 5: Correlation for fine-tuned backbones in MetaWorld. Our proxy demonstrates strong correlation with policy performance when visual backbones are fine-tuned with the policy objective.