Limited Linguistic Diversity in Embodied AI Datasets

TL;DR

The paper tackles the problem that linguistic variation in embodied AI datasets is poorly documented and potentially limits generalization. It introduces three complementary analyses—Duplication/Lexical, Semantic, and Structural Diversity—to quantify language coverage across VLA and robotics corpora, revealing pervasive repetition, limited vocabulary, and shallow syntactic variety in many datasets. Key findings show SCOUT as comparatively more diverse, but most VLA data are dominated by templated, short commands with few negations or conditionals, constraining linguistic grounding. The authors propose practical directions for improvement, including targeted data augmentation, cross-domain language resources, and explicit annotation practices to broaden language coverage and improve evaluation protocols in embodied AI.

Abstract

Language plays a critical role in Vision-Language-Action (VLA) models, yet the linguistic characteristics of the datasets used to train and evaluate these systems remain poorly documented. In this work, we present a systematic dataset audit of several widely used VLA corpora, aiming to characterize what kinds of instructions these datasets actually contain and how much linguistic variety they provide. We quantify instruction language along complementary dimensions-including lexical variety, duplication and overlap, semantic similarity, and syntactic complexity. Our analysis shows that many datasets rely on highly repetitive, template-like commands with limited structural variation, yielding a narrow distribution of instruction forms. We position these findings as descriptive documentation of the language signal available in current VLA training and evaluation data, intended to support more detailed dataset reporting, more principled dataset selection, and targeted curation or augmentation strategies that broaden language coverage.

Figures (20)

• Figure 1: We analyze linguistic diversity in embodied AI datasets across three categories: A.1 Duplication & Lexical diversity, A.2 Semantic diversity, and A.3 Structural diversity.
• Figure 2: (A.2) Verb and direct object co-occurrence frequencies in the manually annotated RT-1 dataset. The heatmap highlights limited verb diversity across plausible actions; for instance, banana is frequently “picked” but never “moved.” The rare verb knock appears mostly with can-shaped objects, despite being equally applicable to others like an upright sponge.
• Figure 3: (A.3) Most frequent POS patterns per dataset on unique sentences and their relative frequency and example.
• Figure 4: (A.3) Percentage of instructions exhibiting four structural phenomena: negation, conditionality, multi-step sequencing, and cyclic repetition. For datasets containing fewer than 600 unique sentences, annotations were performed manually. For those with more than 600 unique sentences, annotations were generated using an automated pipeline. Standard error bars reflect labeling uncertainty estimated from a manually reviewed subset of 500 randomly sampled commands per dataset.
• Figure 5: (A.1) Distribution of command lengths across six examined EAI datasets.