Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lightweight Visual Reasoning for Socially-Aware Robots

Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas, Katie Winkle, Didem Gürdür Broo, Ginevra Castellano

TL;DR

A lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs and evaluates this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (the authors' HRI dataset).

Abstract

Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by $3.3\%$ (less distance), $+0.057$ description score, and $+2.93\%$ accuracy, with less than $3\%$ extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains $+0.111,+0.055$ and $+10.81\%,+4.79\%$ on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics

Lightweight Visual Reasoning for Socially-Aware Robots

TL;DR

A lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs and evaluates this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (the authors' HRI dataset).

Abstract

Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by (less distance), description score, and accuracy, with less than extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains and on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics
Paper Structure (28 sections, 1 equation, 2 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 1 equation, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Vision models in robotic systems
  4. Visual reasoning
  5. Methodology
  6. Architecture choice
  7. Visual reasoning module
  8. Training strategy
  9. Training data
  10. Inference
  11. Optimization and infrastructure
  12. Evaluation
  13. Human intention recognition dataset
  14. Annotation and benchmark adaptation
  15. Ablations and variations
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Overview of the Visual Reasoning approach: A module connects an LLM's hidden states for image tokens back to the vision encoder through a gated MLP, creating a reasoning loop between text and vision. Training uses a two-pass strategy: the first pass extracts reasoning features from the LLM, and the second integrates them into the image encoding, reinterpreting the visual content in light of textual context and reasoning. Detailed description shown in Algorithm \ref{['alg:training']}. A sample of the custom dataset on intention recognition during human-robot interactions is shown in the bottom left.
  • Figure 2: Examples of the datasets use for evaluation. The Mementos-Robotics dataset wang2024mementos provides sequential images with scene descriptions; the Navigation benchmark puig2023habitat3 contains robot trajectories in simulation; and our human-intention recognition dataset captures interactions between humans and a social robot.