Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mobile Manipulation Instruction Generation from Multiple Images with Automatic Metric Enhancement

Kei Katsumata, Motonari Kambara, Daichi Yashima, Ryosuke Korekata, Komei Sugiura

TL;DR

This work tackles free-form mobile manipulation instruction generation from two images: a target object and a receptacle. It introduces Triplet Qformer to align two visual streams with text anchors and a human-centric calibration phase (HCCP) that optimizes a hybrid reward via HCCT loss combining learning-based and $n$-gram metrics. On HM3D-FC, the approach outperforms representative multimodal models across standard metrics, and data augmentation using generated instructions improves downstream IROV-FC performance and real-robot success in physical tests. The method provides a scalable path toward richer, more reliable robot instructions grounded in multi-image understanding, with practical implications for service robotics and assistive applications.

Abstract

We consider the problem of generating free-form mobile manipulation instructions based on a target object image and receptacle image. Conventional image captioning models are not able to generate appropriate instructions because their architectures are typically optimized for single-image. In this study, we propose a model that handles both the target object and receptacle to generate free-form instruction sentences for mobile manipulation tasks. Moreover, we introduce a novel training method that effectively incorporates the scores from both learning-based and n-gram based automatic evaluation metrics as rewards. This method enables the model to learn the co-occurrence relationships between words and appropriate paraphrases. Results demonstrate that our proposed method outperforms baseline methods including representative multimodal large language models on standard automatic evaluation metrics. Moreover, physical experiments reveal that using our method to augment data on language instructions improves the performance of an existing multimodal language understanding model for mobile manipulation.

Mobile Manipulation Instruction Generation from Multiple Images with Automatic Metric Enhancement

TL;DR

This work tackles free-form mobile manipulation instruction generation from two images: a target object and a receptacle. It introduces Triplet Qformer to align two visual streams with text anchors and a human-centric calibration phase (HCCP) that optimizes a hybrid reward via HCCT loss combining learning-based and -gram metrics. On HM3D-FC, the approach outperforms representative multimodal models across standard metrics, and data augmentation using generated instructions improves downstream IROV-FC performance and real-robot success in physical tests. The method provides a scalable path toward richer, more reliable robot instructions grounded in multi-image understanding, with practical implications for service robotics and assistive applications.

Abstract

We consider the problem of generating free-form mobile manipulation instructions based on a target object image and receptacle image. Conventional image captioning models are not able to generate appropriate instructions because their architectures are typically optimized for single-image. In this study, we propose a model that handles both the target object and receptacle to generate free-form instruction sentences for mobile manipulation tasks. Moreover, we introduce a novel training method that effectively incorporates the scores from both learning-based and n-gram based automatic evaluation metrics as rewards. This method enables the model to learn the co-occurrence relationships between words and appropriate paraphrases. Results demonstrate that our proposed method outperforms baseline methods including representative multimodal large language models on standard automatic evaluation metrics. Moreover, physical experiments reveal that using our method to augment data on language instructions improves the performance of an existing multimodal language understanding model for mobile manipulation.

Paper Structure

This paper contains 21 sections, 6 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Language Processing for Robotics
  4. Data Augmentation
  5. Image Captioning
  6. Problem Statements
  7. Proposed Method
  8. Multi Image Feature Generator
  9. Region encoder branch
  10. Grid encoder branch
  11. Triplet Qformer
  12. LLM Decoder
  13. Training Stages
  14. Experimental Settings
  15. Experimental Results
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Overview of our method. Our method generates a mobile manipulation instruction for a given target object image and receptacle image.
  • Figure 2: Overall architecture of the proposed model. ⓒ, SVE, MVE, MLLM, and FC represent concatenation, single-modal visual encoder, multimodal visual encoder, multimodal large language model, and fully connected layer, respectively.
  • Figure 3: The details of MCFormer. Img Trm., Txt Trm., FFN and MHA represent image transformer, text transformer, feed-forward network and multi-head attention, respectively. Here, we define $\bm{H}$ as $\bm{H} = \left\{ \bm{h}_1, ..., \bm{h}_N \right\}.$
  • Figure 4: Successed samples with the proposed method. (a) and (b) show the target object image and the receptacle image, respectively. The bounding box indicates the target object and receptacle that are included in the references.
  • Figure 5: A failed samples obtained by the proposed method. (a) Target object image and (b) receptacle image, respectively.