CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

TL;DR

CALVIN introduces an open-source simulated benchmark for long-horizon language-conditioned robotic manipulation across 34 tasks in four environments, enabling zero-shot generalization to unseen scenes. It integrates multimodal perception, 7-DOF continuous control, and unconstrained natural language instructions, with unstructured play data and procedurally generated language annotations. A multicontext imitation learning baseline demonstrates limited success on long-horizon tasks, underscoring the need for improved language grounding and multimodal representation learning. The benchmark provides standardized evaluation protocols and sensor configurations to drive progress toward scalable language-driven robotics in realistic settings.

Abstract

General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this paper, we present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments and objects. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

Figures (8)

• Figure 2: Observation and action spaces supported by .
• Figure 3: supports a range of sensors commonly utilized for visuomotor control: RGB-D images from both a static and a gripper camera, proprioceptive information, and vision-based tactile sensing (bottom-left).
• Figure 4: Example crowd-sourced natural language instructions to specify manipulation tasks in .
• Figure 5: Example long-horizon language tasks sequences evaluated in . We show the abbreviated subtask names instead of the full language annotations due to space constraint.
• Figure 6: List of all 34 tasks with their respective success criteria.