Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation

Chuhao Jin, Wenhui Tan, Jiange Yang, Bei Liu, Ruihua Song, Limin Wang, Jianlong Fu

TL;DR

The paper tackles the challenge of teaching robots high-level cognitive planning for complex manipulation by introducing AlphaBlock, a large-scale automatically collected dataset, and CogLoop, a multimodal planning framework that fuses vision, language, and embodied feedback. AlphaBlock enables end-to-end learning of sub-task planning grounded in real observations, while CogLoop uses a vision adapter and a Q-former to align visual inputs with an LLM, enabling autoregressive plan generation and controlled execution via a frozen policy. The approach yields significant performance gains over language-only baselines in simulated tasks and demonstrates real-world deployment potential on a robot arm, underscoring the value of closed-loop, vision-informed planning for robust robotic cognition. The work lays a foundation for scalable, cognitively capable robots that can understand and execute high-level instructions with minimal human supervision.

Abstract

We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks. These tasks often involve complex multi-step reasoning, presenting significant challenges due to the limited paired data connecting human instructions (e.g., making a smiley face) and robot actions (e.g., end-effector movement). Existing approaches relieve this challenge by adopting an open-loop paradigm decomposing high-level instructions into simple sub-task plans, and executing them step-by-step using low-level control models. However, these approaches are short of instant observations in multi-step reasoning, leading to sub-optimal results. To address this issue, we propose to automatically collect a cognitive robot dataset by Large Language Models (LLMs). The resulting dataset AlphaBlock consists of 35 comprehensive high-level tasks of multi-step text plans and paired observation sequences. To enable efficient data acquisition, we employ elaborated multi-round prompt designs that effectively reduce the burden of extensive human involvement. We further propose a closed-loop multi-modal embodied planning model that autoregressively generates plans by taking image observations as input. To facilitate effective learning, we leverage MiniGPT-4 with a frozen visual encoder and LLM, and finetune additional vision adapter and Q-former to enable fine-grained spatial perception for manipulation tasks. We conduct experiments to verify the superiority over existing open and closed-loop methods, and achieve a significant increase in success rate by 21.4% and 14.5% over ChatGPT and GPT-4 based robot tasks. Real-world demos are shown in https://www.youtube.com/watch?v=ayAzID1_qQk .

AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation

TL;DR

The paper tackles the challenge of teaching robots high-level cognitive planning for complex manipulation by introducing AlphaBlock, a large-scale automatically collected dataset, and CogLoop, a multimodal planning framework that fuses vision, language, and embodied feedback. AlphaBlock enables end-to-end learning of sub-task planning grounded in real observations, while CogLoop uses a vision adapter and a Q-former to align visual inputs with an LLM, enabling autoregressive plan generation and controlled execution via a frozen policy. The approach yields significant performance gains over language-only baselines in simulated tasks and demonstrates real-world deployment potential on a robot arm, underscoring the value of closed-loop, vision-informed planning for robust robotic cognition. The work lays a foundation for scalable, cognitively capable robots that can understand and execute high-level instructions with minimal human supervision.

Abstract

We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks. These tasks often involve complex multi-step reasoning, presenting significant challenges due to the limited paired data connecting human instructions (e.g., making a smiley face) and robot actions (e.g., end-effector movement). Existing approaches relieve this challenge by adopting an open-loop paradigm decomposing high-level instructions into simple sub-task plans, and executing them step-by-step using low-level control models. However, these approaches are short of instant observations in multi-step reasoning, leading to sub-optimal results. To address this issue, we propose to automatically collect a cognitive robot dataset by Large Language Models (LLMs). The resulting dataset AlphaBlock consists of 35 comprehensive high-level tasks of multi-step text plans and paired observation sequences. To enable efficient data acquisition, we employ elaborated multi-round prompt designs that effectively reduce the burden of extensive human involvement. We further propose a closed-loop multi-modal embodied planning model that autoregressively generates plans by taking image observations as input. To facilitate effective learning, we leverage MiniGPT-4 with a frozen visual encoder and LLM, and finetune additional vision adapter and Q-former to enable fine-grained spatial perception for manipulation tasks. We conduct experiments to verify the superiority over existing open and closed-loop methods, and achieve a significant increase in success rate by 21.4% and 14.5% over ChatGPT and GPT-4 based robot tasks. Real-world demos are shown in https://www.youtube.com/watch?v=ayAzID1_qQk .
Paper Structure (40 sections, 7 equations, 9 figures, 7 tables)

This paper contains 40 sections, 7 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Instruction Following Robot Control
  4. Robot Task Planning with LLMs
  5. Approach
  6. Problem Setup
  7. Automatic Data Collection for Real-time Planning and Control
  8. Task Definition
  9. Prompt Design for Placement Collection
  10. Self-Verification to Ensure Data Quality
  11. CogLoop as Robot Planner
  12. Experiment
  13. Training Details
  14. Experimental Settings
  15. Compared Methods
  16. ...and 25 more sections

Figures (9)

  • Figure 1: Planner model paradigms. (a) Open-loop models (SayCan-style saycan) conduct planning and control separately. (b) Closed-loop models update plans with observation in language (Text2Motion-style text2motion). (c) We infuse more fine-grained visual observation into LLM to update planning.
  • Figure 2: Examples of block placement for high-level task from our AlphaBlock dataset. We show each capital letter of "NeurIPS", and "a smiley face". The robot arm is placed at random positions.
  • Figure 3: CogLoop consists of three main components. 1) A pre-trained ViT that serves as efficient feature extractors. 2) Parameter-efficient tuning module includes a Vision Adapter and a combined Q-former with a projector to align multi-stage visual features with language space. 3) A frozen LLM which processes the task description and visual observation to reason out sub-task plans. The plans are subsequently applied to a frozen Policy Model to generate actionable steps, which are then applied in embodied environments to obtain the next observation state. The dashed line indicates the next plan after generating the previous one. [Best viewed in color]
  • Figure 4: Examples of prompts and response from LLM in open-loop and closed-loop settings. We highlight importance in blue and difference in red. [Better viewed in color.]
  • Figure 5: The prompt for GPT-4 to generate the position of the given layout letter "K".
  • ...and 4 more figures