Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinker: A vision-language foundation model for embodied intelligence

Baiyu Pan, Daqin Luo, Junpeng Yang, Jiyuan Wang, Yixuan Zhang, Hailin Shi, Jichao Jiao

TL;DR

Thinker tackles a critical gap in robotics-oriented vision-language models by anchoring understanding in egocentric spatial, temporal, and grounding capabilities. It achieves this through a two-pronged approach: (i) constructing large, diverse, robot-focused datasets for visual grounding, ego-view reasoning, and long-horizon planning, and (ii) a two-stage training strategy that first builds embodied perception and reasoning, then fine-tunes for downstream planning, including industrial tasks. The model, Thinker-7B, demonstrates state-of-the-art performance on RobovQA and Egoplan-bench2 benchmarks, illustrating strong video-perception and task-planning abilities across diverse scenarios. These results indicate promising practical impact for robotic perception and planning, enabling more reliable, end-to-end vision-language systems in real-world embodied intelligence settings.

Abstract

When large vision-language models are applied to the field of robotics, they encounter problems that are simple for humans yet error-prone for models. Such issues include confusion between third-person and first-person perspectives and a tendency to overlook information in video endings during temporal reasoning. To address these challenges, we propose Thinker, a large vision-language foundation model designed for embodied intelligence. We tackle the aforementioned issues from two perspectives. Firstly, we construct a large-scale dataset tailored for robotic perception and reasoning, encompassing ego-view videos, visual grounding, spatial understanding, and chain-of-thought data. Secondly, we introduce a simple yet effective approach that substantially enhances the model's capacity for video comprehension by jointly incorporating key frames and full video sequences as inputs. Our model achieves state-of-the-art results on two of the most commonly used benchmark datasets in the field of task planning.

Thinker: A vision-language foundation model for embodied intelligence

TL;DR

Thinker tackles a critical gap in robotics-oriented vision-language models by anchoring understanding in egocentric spatial, temporal, and grounding capabilities. It achieves this through a two-pronged approach: (i) constructing large, diverse, robot-focused datasets for visual grounding, ego-view reasoning, and long-horizon planning, and (ii) a two-stage training strategy that first builds embodied perception and reasoning, then fine-tunes for downstream planning, including industrial tasks. The model, Thinker-7B, demonstrates state-of-the-art performance on RobovQA and Egoplan-bench2 benchmarks, illustrating strong video-perception and task-planning abilities across diverse scenarios. These results indicate promising practical impact for robotic perception and planning, enabling more reliable, end-to-end vision-language systems in real-world embodied intelligence settings.

Abstract

When large vision-language models are applied to the field of robotics, they encounter problems that are simple for humans yet error-prone for models. Such issues include confusion between third-person and first-person perspectives and a tendency to overlook information in video endings during temporal reasoning. To address these challenges, we propose Thinker, a large vision-language foundation model designed for embodied intelligence. We tackle the aforementioned issues from two perspectives. Firstly, we construct a large-scale dataset tailored for robotic perception and reasoning, encompassing ego-view videos, visual grounding, spatial understanding, and chain-of-thought data. Secondly, we introduce a simple yet effective approach that substantially enhances the model's capacity for video comprehension by jointly incorporating key frames and full video sequences as inputs. Our model achieves state-of-the-art results on two of the most commonly used benchmark datasets in the field of task planning.
Paper Structure (21 sections, 2 figures, 2 tables)

This paper contains 21 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Training Data
  3. Visual Grounding Data
  4. Ego-View Reasoning Data
  5. Robotic Manipulation Planning Data
  6. Industrial Task Planning Data
  7. Thinker Models
  8. Model architecture
  9. Training Strategy
  10. Stage-1: Building Embodied Capabilities
  11. Stage-2: Downstream Task Fine-Tuning
  12. Infrastructures
  13. Large-Scale Multi-Task Training Infrastructure
  14. Inference Infrastructure for Fine-Tuned Model
  15. Fault Tolerance and Monitoring
  16. ...and 6 more sections

Figures (2)

  • Figure 1: This figure illustrates the distribution of our crafted training datasets, that we categorized in to four types: visual grounding, ego-view, planning, industrial.
  • Figure 2: Our model supports images, videos, and complex language instructions. Video sequences, along with their last frames, are processed through a vision encoder and an adapter. All input tokens are subsequently concatenated and fed into the Thinker decoder.