Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

J-ORA: A Framework and Multimodal Dataset for Japanese Object Identification, Reference, Action Prediction in Robot Perception

Jesse Atuhurra, Hidetaka Kamigaito, Taro Watanabe, Koichiro Yoshino

TL;DR

J-ORA introduces a rich attribute-annotated multimodal dataset to advance Japanese robot perception across object identification, reference resolution, and action prediction. By extending J-CRe3 with a detailed attribute template and dynamic scene annotations, it enables end-to-end multimodal perception models trained on egocentric, indoor data. Experimental results show attribute information improves performance, but there remains a gap between proprietary and open VLMs and affordance understanding varies by object. The work provides a scalable framework for semi-automatic dataset expansion in dynamic, real-world environments.

Abstract

We introduce J-ORA, a novel multimodal dataset that bridges the gap in robot perception by providing detailed object attribute annotations within Japanese human-robot dialogue scenarios. J-ORA is designed to support three critical perception tasks, object identification, reference resolution, and next-action prediction, by leveraging a comprehensive template of attributes (e.g., category, color, shape, size, material, and spatial relations). Extensive evaluations with both proprietary and open-source Vision Language Models (VLMs) reveal that incorporating detailed object attributes substantially improves multimodal perception performance compared to without object attributes. Despite the improvement, we find that there still exists a gap between proprietary and open-source VLMs. In addition, our analysis of object affordances demonstrates varying abilities in understanding object functionality and contextual relationships across different VLMs. These findings underscore the importance of rich, context-sensitive attribute annotations in advancing robot perception in dynamic environments. See project page at https://jatuhurrra.github.io/J-ORA/.

J-ORA: A Framework and Multimodal Dataset for Japanese Object Identification, Reference, Action Prediction in Robot Perception

TL;DR

J-ORA introduces a rich attribute-annotated multimodal dataset to advance Japanese robot perception across object identification, reference resolution, and action prediction. By extending J-CRe3 with a detailed attribute template and dynamic scene annotations, it enables end-to-end multimodal perception models trained on egocentric, indoor data. Experimental results show attribute information improves performance, but there remains a gap between proprietary and open VLMs and affordance understanding varies by object. The work provides a scalable framework for semi-automatic dataset expansion in dynamic, real-world environments.

Abstract

We introduce J-ORA, a novel multimodal dataset that bridges the gap in robot perception by providing detailed object attribute annotations within Japanese human-robot dialogue scenarios. J-ORA is designed to support three critical perception tasks, object identification, reference resolution, and next-action prediction, by leveraging a comprehensive template of attributes (e.g., category, color, shape, size, material, and spatial relations). Extensive evaluations with both proprietary and open-source Vision Language Models (VLMs) reveal that incorporating detailed object attributes substantially improves multimodal perception performance compared to without object attributes. Despite the improvement, we find that there still exists a gap between proprietary and open-source VLMs. In addition, our analysis of object affordances demonstrates varying abilities in understanding object functionality and contextual relationships across different VLMs. These findings underscore the importance of rich, context-sensitive attribute annotations in advancing robot perception in dynamic environments. See project page at https://jatuhurrra.github.io/J-ORA/.
Paper Structure (21 sections, 5 equations, 4 figures, 7 tables)

This paper contains 21 sections, 5 equations, 4 figures, 7 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Object Attribute Datasets
  4. Multimodal Dialogue Datasets
  5. Multimodal Perception
  6. Robot Action Planning
  7. PRELIMINARIES FOR END-TO-END MULTIMODAL PERCEPTION
  8. J-ORA DATASET
  9. J-CRe3 Dataset
  10. Object Attributes
  11. Changes in the Visual Scene during Dialogue
  12. Object Attribute Annotation
  13. Data Statistics
  14. EXPERIMENTS
  15. Experimental Settings
  16. ...and 6 more sections

Figures (4)

  • Figure 1: We investigate the contribution of object attributes towards the robot's perception, and introduce three perception tasks to facilitate the analysis.
  • Figure 2: Left: We develop a template of object attributes and use it to annotate objects in Japanese, creating J-ORA. A highlighted instance from J-ORA at the bottom shows two objects (backpack and laptop). The scene comprises the robot's egocentric view. Right: Our multimodal end-to-end perception system, based on a VLM, consists of three tasks that are performed simultaneously. System inputs include images/videos, dialogue utterances, and prompts. The human utterance above translates into 'Place the tissue box in the green backpack.'
  • Figure 3: Changes that occur in the scene during dialogue. We have shown image IDs for reference to image sequences.
  • Figure 4: Fine-tuned open VLM Accuracy, without attributes.