A Taxonomy for Evaluating Generalist Robot Policies

TL;DR

The paper addresses the fragmentation of generalization evaluations in robot manipulation by introducing STAR-Gen, a comprehensive taxonomy that organizes perturbations by vision, language, and action modalities. It then instantiates a real-world benchmark, BridgeV2-$STAR$, to systematically measure 13 axes of generalization across multiple VLA models, revealing persistent semantic-generalization gaps and informing model and data-design choices. Key findings show that scaling data and larger language backbones help some axes, while semantic generalization remains challenging; vector-quantized action chunking and targeted co-training can provide broader gains, though with mixed effects. The work offers practical guidelines for constructing more thorough benchmarks and collecting targeted data to drive progress toward deployable, generalist robot policies.

Abstract

Machine learning for robotics promises to unlock generalization to novel tasks and environments. Guided by this promise, many recent works have focused on scaling up robot data collection and developing larger, more expressive policies to achieve this. But how do we measure progress towards this goal of policy generalization in practice? Evaluating and quantifying generalization is the Wild West of modern robotics, with each work proposing and measuring different types of generalization in their own, often difficult to reproduce, settings. In this work, our goal is (1) to outline the forms of generalization we believe are important in robot manipulation in a comprehensive and fine-grained manner, and (2) to provide reproducible guidelines for measuring these notions of generalization. We first propose STAR-Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalization. We discuss how our taxonomy encompasses most prior notions of generalization in robotics. Next, we instantiate STAR-Gen with a concrete real-world benchmark based on the widely-used Bridge V2 dataset. We evaluate a variety of state-of-the-art models on this benchmark to demonstrate the utility of our taxonomy in practice. Our taxonomy of generalization can yield many interesting insights into existing models: for example, we observe that current vision-language-action models struggle with various types of semantic generalization, despite the promise of pre-training on internet-scale language datasets. We believe STAR-Gen and our guidelines can improve the dissemination and evaluation of progress towards generalization in robotics, which we hope will guide model design and future data collection efforts. We provide videos and demos at our website stargen-taxonomy.github.io.

Figures (7)

• Figure 1: Visualization of $\bigstar$-Gen applied to the example base task "put carrot on plate". $\bigstar$-Gen is structured around perturbations to the three modalities of visuo-lingual policies (visual, semantic, behavioral), with consideration for each possible combination of each modality. We refer to these combinations as categories. For each category (colored sectors), we group different perturbations of that category into axes of generalization (light colored boxes). We provide some example perturbations.
• Figure 2: BridgeV2-$\bigstar$ main results. We report aggregated success rates for each model and axis, including in-distribution (ID).
• Figure 3: We investigate different VLA design decisions to assess their impact on generalization. We report success rates across all trials for each model and axis. (a) Scaling robot dataset size and diversity can improve multiple axes of generalization. (b) Using a larger LLM backbone can improve semantic generalization, but only to a limited extent. (c) VQA co-training can improve some axes, but has a mixed effect on semantic axes. (d) Vector quantized action chunking can improve multiple axes.
• Figure 4: We find that co-fine-tuning on in-domain data for our base tasks significantly improves both in-distribution performance of OpenVLA on our base tasks, as well as generalization for many of our axes.
• Figure 5: Example of using our demonstration to generate New Object perturbations for the base task "pick up the AAA battery".