Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications

Nghia Nguyen, Minh Nhat Vu, Tung D. Ta, Baoru Huang, Thieu Vo, Ngan Le, Anh Nguyen

TL;DR

By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts, and outperforms other CLIP-based models across various language-driven robotic tasks.

Abstract

Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309,433 videos (~7.4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.

Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications

TL;DR

By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts, and outperforms other CLIP-based models across various language-driven robotic tasks.

Abstract

Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309,433 videos (~7.4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.
Paper Structure (14 sections, 10 equations, 6 figures, 6 tables)

This paper contains 14 sections, 10 equations, 6 figures, 6 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. METHODOLOGY
  4. Motivation
  5. Data Preparation
  6. Robotic-CLIP
  7. Experiments
  8. Robotic-CLIP in Language-driven Grasp Detection
  9. Robotic-CLIP in Policy Learning
  10. Robotic-CLIP in Robot Navigation
  11. Ablation Study
  12. Robotic Experiment
  13. Limitation
  14. Discussion

Figures (6)

  • Figure 1: The comparison between: (a) CLIP radford2021learning, (b) Alpha-CLIP sun2024alpha, and (c) our Robotic-CLIP. CLIP aligns scene descriptions with images, while Alpha-CLIP adds object masks. Our method aligns paired frames with action descriptions and masks to capture the action in robotic tasks.
  • Figure 2: The dataset preparation pipeline (a) and our fine-tuning pipeline (b).
  • Figure 3: Language-drive grasp detection results of LLGD nguyen2024lightweight using different CLIP models.
  • Figure 4: Results of policy learning experiments. With the text prompt input "Open Microwave", we compare the robot actions generated by the baseline LIVma2023liv with our method and other CLIP-based models. The top row represents CLIP radford2021learning, the middle row corresponds to Alpha-CLIP sun2024alpha, and the bottom row illustrates our proposed Robotic-CLIP method.
  • Figure 5: Text-frame similarity analysis.
  • ...and 1 more figures