Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction

David Sobrín-Hidalgo, Miguel Ángel González-Santamarta, Ángel Manuel Guerrero-Higueras, Francisco Javier Rodríguez-Lera, Vicente Matellán-Olivera

TL;DR

The paper addresses the challenge of explaining autonomous robot actions in human-robot interaction when visual context matters. It extends a prior LLM-based log-explanation system by incorporating a Vision-Language Model to interpret onboard camera data and fuse it with textual logs via a Retrieval Augmented Generation pipeline. The approach is validated in a simulated navigation task with an unexpected obstacle, demonstrating that multimodal explanations can more accurately identify events (e.g., obstacles) and justify route changes. The findings indicate that vision-enabled explanations enhance precision and contextual understanding, contributing to more trustworthy autonomous systems. This multimodal explainability framework has practical implications for HRI across ROS 2-based robots and can be extended to additional modalities in future work.

Abstract

This paper presents an improved system based on our prior work, designed to create explanations for autonomous robot actions during Human-Robot Interaction (HRI). Previously, we developed a system that used Large Language Models (LLMs) to interpret logs and produce natural language explanations. In this study, we expand our approach by incorporating Vision-Language Models (VLMs), enabling the system to analyze textual logs with the added context of visual input. This method allows for generating explanations that combine data from the robot's logs and the images it captures. We tested this enhanced system on a basic navigation task where the robot needs to avoid a human obstacle. The findings from this preliminary study indicate that adding visual interpretation improves our system's explanations by precisely identifying obstacles and increasing the accuracy of the explanations provided.

Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction

TL;DR

The paper addresses the challenge of explaining autonomous robot actions in human-robot interaction when visual context matters. It extends a prior LLM-based log-explanation system by incorporating a Vision-Language Model to interpret onboard camera data and fuse it with textual logs via a Retrieval Augmented Generation pipeline. The approach is validated in a simulated navigation task with an unexpected obstacle, demonstrating that multimodal explanations can more accurately identify events (e.g., obstacles) and justify route changes. The findings indicate that vision-enabled explanations enhance precision and contextual understanding, contributing to more trustworthy autonomous systems. This multimodal explainability framework has practical implications for HRI across ROS 2-based robots and can be extended to additional modalities in future work.

Abstract

This paper presents an improved system based on our prior work, designed to create explanations for autonomous robot actions during Human-Robot Interaction (HRI). Previously, we developed a system that used Large Language Models (LLMs) to interpret logs and produce natural language explanations. In this study, we expand our approach by incorporating Vision-Language Models (VLMs), enabling the system to analyze textual logs with the added context of visual input. This method allows for generating explanations that combine data from the robot's logs and the images it captures. We tested this enhanced system on a basic navigation task where the robot needs to avoid a human obstacle. The findings from this preliminary study indicate that adding visual interpretation improves our system's explanations by precisely identifying obstacles and increasing the accuracy of the explanations provided.
Paper Structure (7 sections, 4 figures)

This paper contains 7 sections, 4 figures.

Table of Contents

  1. INTRODUCTION
  2. MATERIALS AND METHODS
  3. Explanation generation with LLMs
  4. Enhancing explanation capabilities through VLMS
  5. Evaluation
  6. RESULTS & DISCUSSION
  7. CONCLUSIONS

Figures (4)

  • Figure 1: New workflow of the explainability system sobrínhidalgo2024explaining. The incorporation of the VLM along with the corresponding changes are reflected in the Storage stage.
  • Figure 2: New ROS 2-based Explainability System implementation sobrínhidalgo2024explaining.
  • Figure 3: Frame evaluated to provide explanation.
  • Figure 4: Robot trajectories during experiment. S indicates the starting point and G is the goal point. The green line indicates the original trajectories, while the red line indicates the new trajectories after replanning.