Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CARE: Multi-Task Pretraining for Latent Continuous Action Representation in Robot Control

Jiaqi Shi, Xulong Zhang, Xiaoyang Qu, Jianzong Wang

TL;DR

CARE tackles the reliance on explicit action labels for pretraining Visual-Language-Action models in robotics by introducing a multi-task pretraining framework that learns continuous latent actions from video-text pairs. It replaces separate latent-action modules with a latent Vision-Language Model and adds two decoders for frame prediction and 2D keypoint trajectory prediction, guided by an Uncertainty-Weighted Loss to train a continuous latent action $z$. A lightweight action head, trained on a small labeled dataset with LoRA fine-tuning, maps latent actions to real robot actions. Evaluation on LIBERO across robot and human datasets shows CARE outperforms label-free baselines and remains competitive with label-based methods, while achieving stronger interpretability and reduced shortcut learning. This approach demonstrates scalable, weakly supervised pretraining for VLA in robotics with practical benefits in control and transfer.

Abstract

Recent advances in Vision-Language-Action (VLA) models have shown promise for robot control, but their dependence on action supervision limits scalability and generalization. To address this challenge, we introduce CARE, a novel framework designed to train VLA models for robotic task execution. Unlike existing methods that depend on action annotations during pretraining, CARE eliminates the need for explicit action labels by leveraging only video-text pairs. These weakly aligned data sources enable the model to learn continuous latent action representations through a newly designed multi-task pretraining objective. During fine-tuning, a small set of labeled data is used to train the action head for control. Experimental results across various simulation tasks demonstrate CARE's superior success rate, semantic interpretability, and ability to avoid shortcut learning. These results underscore CARE's scalability, interpretability, and effectiveness in robotic control with weak supervision.

CARE: Multi-Task Pretraining for Latent Continuous Action Representation in Robot Control

TL;DR

CARE tackles the reliance on explicit action labels for pretraining Visual-Language-Action models in robotics by introducing a multi-task pretraining framework that learns continuous latent actions from video-text pairs. It replaces separate latent-action modules with a latent Vision-Language Model and adds two decoders for frame prediction and 2D keypoint trajectory prediction, guided by an Uncertainty-Weighted Loss to train a continuous latent action . A lightweight action head, trained on a small labeled dataset with LoRA fine-tuning, maps latent actions to real robot actions. Evaluation on LIBERO across robot and human datasets shows CARE outperforms label-free baselines and remains competitive with label-based methods, while achieving stronger interpretability and reduced shortcut learning. This approach demonstrates scalable, weakly supervised pretraining for VLA in robotics with practical benefits in control and transfer.

Abstract

Recent advances in Vision-Language-Action (VLA) models have shown promise for robot control, but their dependence on action supervision limits scalability and generalization. To address this challenge, we introduce CARE, a novel framework designed to train VLA models for robotic task execution. Unlike existing methods that depend on action annotations during pretraining, CARE eliminates the need for explicit action labels by leveraging only video-text pairs. These weakly aligned data sources enable the model to learn continuous latent action representations through a newly designed multi-task pretraining objective. During fine-tuning, a small set of labeled data is used to train the action head for control. Experimental results across various simulation tasks demonstrate CARE's superior success rate, semantic interpretability, and ability to avoid shortcut learning. These results underscore CARE's scalability, interpretability, and effectiveness in robotic control with weak supervision.
Paper Structure (12 sections, 1 equation, 2 figures, 3 tables)

This paper contains 12 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. Latent Vision-Language Model
  4. Continuous Latent Action with Multi-task Learning
  5. Fine-tuning VLA with action labels
  6. EXPERIMENT
  7. Setups
  8. Main Results
  9. Ablation Study on Interpretability
  10. Ablation Study on Shortcut Learning
  11. Ablation Study on Multi-task Learning
  12. Conclusion

Figures (2)

  • Figure 1: Overview of CARE. Compared to (a) the VLA training approach that requires a large number of action labels, (b) our proposed multitask learning-based pre-training method, CARE, uses only instruction-image pairs as input. Without relying on action labels, it still enables the VLM to predict potential actions.
  • Figure 2: Datasets of pre-training stage and post-training stage.