CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision

TL;DR

This work introduces CLIP-RT, a discriminative vision-language-action model that learns language-conditioned visuomotor policies from natural language supervision. A two-stage data collection framework—language-based teleoperation and stochastic trajectory augmentation (STA)—enables non-experts to gather diverse robot demonstrations, which are used to pretrain CLIP-RT on the Open X-Embodiment dataset and fine-tune in-domain. Empirical results show CLIP-RT outperforms OpenVLA on novel real-world tasks and attains strong few-shot generalization, human collaboration, and GPT-assisted planning capabilities, with competitive performance in simulation via the LIBERO benchmark and far superior runtime efficiency. The approach significantly increases accessibility and scalability of robot learning by leveraging natural language as supervision and a lightweight, CLIP-based architecture.

Abstract

Teaching robots desired skills in real-world environments remains challenging, especially for non-experts. A key bottleneck is that collecting robotic data often requires expertise or specialized hardware, limiting accessibility and scalability. We posit that natural language offers an intuitive and accessible interface for robot learning. To this end, we study two aspects: (1) enabling non-experts to collect robotic data through natural language supervision (e.g., "move the arm to the right") and (2) training robot policies directly from this supervision. Specifically, we introduce a data collection framework that collects robot demonstrations based on natural language supervision and further augments these demonstrations. We then present CLIP-RT, a new vision-language-action (VLA) model that learns language-conditioned visuomotor policies from this supervision. CLIP-RT adapts the pretrained CLIP model and learns to predict language-based motion primitives via contrastive imitation learning. We train CLIP-RT on the Open X-Embodiment dataset and finetune it on in-domain data collected by our framework. In real-world evaluations, CLIP-RT demonstrates strong capabilities in learning novel manipulation skills, outperforming OpenVLA (7B parameters) by 24% in average success rates, while using 7x fewer parameters (1B). We further assess CLIP-RT's capabilities in few-shot generalization and collaborative scenarios involving large pretrained models or humans. In simulated environments, CLIP-RT also yields strong performance, achieving a 93.1% average success rate on the LIBERO benchmark with an inference throughput of 163 Hz.

Figures (19)

• Figure 1: Overview of language-guided teleoperation.
• Figure 2: Overview of CLIP-RT. CLIP-RT learns to optimize the pairwise similarity between the context and natural language supervision through contrastive imitation learning. At test time, CLIP-RT predicts the language-based motion primitive with the highest similarity from a list of language motions. We append a simple text prompt to instructions: What motion should the robot arm perform to complete the instruction $\left\{\text{instruction}\right\}$?
• Figure 3: A simplified 2D example of stochastic trajectory augmentation (STA). (a): a demonstration trajectory from the start $s$ to the endpoint $e$, passing through a waypoint $w_1$. (b): a sampled trajectory generated by the diversification phase. (c)-(e): a visualization of the recovery phase.
• Figure 4: Success rates on 9 Common tasks (top) and 9 Novel tasks (bottom). We conduct experiments using all compared methods on Common tasks and three models (CLIP-RT, OpenVLA and CLIP-RT-Action) on Novel Tasks. The success rate for each task is measured by averaging the results of ten trials. Average success rates of all tasks are shown on the left for both Common and Novel task sets. Tasks are arranged from left to right based on their average number of steps per episode in the training data. The task on the right indicates that it requires more steps in average compared with the task on the left.
• Figure 5: A comparison of multi-task and single-task policies on Novel tasks. The performance of each task is in Figure \ref{['fig:novel_single']} of Appendix.