What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data

TL;DR

This paper tackles language-conditioned robotic imitation learning from offline, unstructured play data by systematically evaluating and integrating key design choices. It proposes a hierarchical control architecture that uses global plans and a local gripper-frame policy, paired with a multimodal transformer-based latent-plan encoder that yields discrete plan representations. A self-supervised contrastive video-language alignment, a discretized action decoder, and careful optimization including KL balancing drive robust learning, culminating in state-of-the-art results on the CALVIN long-horizon benchmark. The work provides open-source code and models to foster further research toward general-purpose, language-grounded manipulation skills.

Abstract

A long-standing goal in robotics is to build robots that can perform a wide range of daily tasks from perceptions obtained with their onboard sensors and specified only via natural language. While recently substantial advances have been achieved in language-driven robotics by leveraging end-to-end learning from pixels, there is no clear and well-understood process for making various design choices due to the underlying variation in setups. In this paper, we conduct an extensive study of the most critical challenges in learning language conditioned policies from offline free-form imitation datasets. We further identify architectural and algorithmic techniques that improve performance, such as a hierarchical decomposition of the robot control learning, a multimodal transformer encoder, discrete latent plans and a self-supervised contrastive loss that aligns video and language representations. By combining the results of our investigation with our improved model components, we are able to present a novel approach that significantly outperforms the state of the art on the challenging language conditioned long-horizon robot manipulation CALVIN benchmark. We have open-sourced our implementation to facilitate future research in learning to perform many complex manipulation skills in a row specified with natural language. Codebase and trained models available at http://hulc.cs.uni-freiburg.de

Figures (5)

• Figure 1: learns a single 7-DoF language conditioned visuomotor policy from offline, unstructured data that can solve multi-stage, long-horizon robot manipulation tasks. We divide instruction following into learning global plans representing high-level behavior and a local policy conditioned on the plan and the instruction.
• Figure 2: Overview of our architecture to learn language conditioned policies from unstructured data. First the language instructions and the visual observations are encoded. During training a multimodal transformer encodes sequences of observations to learn to recognize and organize high-level behaviors through a posterior. Its temporally contextualized features are provided as input to a contrastive visuo-lingual alignment loss. The plan sampler network receives the initial state and the latent language goal and predicts the distribution over plans for achieving the goal. Both prior and posterior distributions are predicted as a vector of multiple categorical variables and are trained by minimizing their KL divergence. The local policy network receives the latent language instruction, the gripper camera observation and the global latent plan to generate a sequence of relative actions in the gripper camera frame to achieve the goal.
• Figure 3: Performance of our model on the D environment of the CALVIN Challenge and ablation of the key components, across 3 seeded runs. All models receive RGB images from both a static and a gripper camera as a input.
• Figure 4: Performance of our model on the multi environment splits of the CALVIN Challenge across 3 seeded runs.
• Figure 5: t-SNE visualization of the discrete latent plans generated by embedding randomly selected unseen language annotations. Surprisingly, we find that despite not being trained explicitly with task labels, appears to organize its latent plan space functionally. We visualize with the same color functionally similar skills, but use different shapes to distinguish sub-skills.