Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proprioception Enhances Vision Language Model in Generating Captions and Subtask Segmentations for Robot Task

Kanata Suzuki, Shota Shimizu, Tetsuya Ogata

TL;DR

This work interrogates whether Vision Language Models (VLMs) can understand robot motion by incorporating proprioceptive data into prompts to perform captioning and subtask segmentation of robot tasks. The method builds scene captions from image captions and low-level robot state, then summarizes them into a complete task caption and uses embedding-based reasoning to split tasks into subtasks, all in a zero-shot framework. Experiments on Robosuite tasks (Door Opening, Bin Pick-placing) demonstrate that including joint angles and end-effector states improves trajectory-aware captions and yields more intuitive subtask boundaries, though fine-grained temporal changes remain challenging and depend on the segmentation threshold. The study suggests that proprioception enhances VLM capability for robot understanding and points to future integration with imitation learning and LLM-driven motion planning to improve practical robotics systems.

Abstract

From the perspective of future developments in robotics, it is crucial to verify whether foundation models trained exclusively on offline data, such as images and language, can understand the robot motion. In particular, since Vision Language Models (VLMs) do not include low-level motion information from robots in their training datasets, video understanding including trajectory information remains a significant challenge. In this study, we assess two capabilities of VLMs through a video captioning task with low-level robot motion information: (1) automatic captioning of robot tasks and (2) segmentation of a series of tasks. Both capabilities are expected to enhance the efficiency of robot imitation learning by linking language and motion and serve as a measure of the foundation model's performance. The proposed method generates multiple "scene" captions using image captions and trajectory data from robot tasks. The full task caption is then generated by summarizing these individual captions. Additionally, the method performs subtask segmentation by comparing the similarity between text embeddings of image captions. In both captioning tasks, the proposed method aims to improve performance by providing the robot's motion data - joint and end-effector states - as input to the VLM. Simulator experiments were conducted to validate the effectiveness of the proposed method.

Proprioception Enhances Vision Language Model in Generating Captions and Subtask Segmentations for Robot Task

TL;DR

This work interrogates whether Vision Language Models (VLMs) can understand robot motion by incorporating proprioceptive data into prompts to perform captioning and subtask segmentation of robot tasks. The method builds scene captions from image captions and low-level robot state, then summarizes them into a complete task caption and uses embedding-based reasoning to split tasks into subtasks, all in a zero-shot framework. Experiments on Robosuite tasks (Door Opening, Bin Pick-placing) demonstrate that including joint angles and end-effector states improves trajectory-aware captions and yields more intuitive subtask boundaries, though fine-grained temporal changes remain challenging and depend on the segmentation threshold. The study suggests that proprioception enhances VLM capability for robot understanding and points to future integration with imitation learning and LLM-driven motion planning to improve practical robotics systems.

Abstract

From the perspective of future developments in robotics, it is crucial to verify whether foundation models trained exclusively on offline data, such as images and language, can understand the robot motion. In particular, since Vision Language Models (VLMs) do not include low-level motion information from robots in their training datasets, video understanding including trajectory information remains a significant challenge. In this study, we assess two capabilities of VLMs through a video captioning task with low-level robot motion information: (1) automatic captioning of robot tasks and (2) segmentation of a series of tasks. Both capabilities are expected to enhance the efficiency of robot imitation learning by linking language and motion and serve as a measure of the foundation model's performance. The proposed method generates multiple "scene" captions using image captions and trajectory data from robot tasks. The full task caption is then generated by summarizing these individual captions. Additionally, the method performs subtask segmentation by comparing the similarity between text embeddings of image captions. In both captioning tasks, the proposed method aims to improve performance by providing the robot's motion data - joint and end-effector states - as input to the VLM. Simulator experiments were conducted to validate the effectiveness of the proposed method.
Paper Structure (14 sections, 8 figures)

This paper contains 14 sections, 8 figures.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Video Captioning
  4. Segmentation of Robot Motion
  5. PROPOSED METHOD
  6. Caption Generation by VLM with Robot State
  7. Scene Caption Generation
  8. Caption Summarization
  9. Subtask Segmentation
  10. EXPERIMENTS
  11. RESULTS AND DISCUSSION
  12. Caption Generation
  13. Subtask Segmentation
  14. CONCLUSION

Figures (8)

  • Figure 1: Overview of this study. We verified the performance of VLM by generating captions and subtask segmentations of robot motions.
  • Figure 2: Overview of the proposed method.
  • Figure 3: Experimental setup. We verified our method in the Door opening and Bin pick-placing task.
  • Figure 4: Examples of generated sub-captions and summarized caption.
  • Figure 5: The success rates of caption generation task.
  • ...and 3 more figures