Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Large Language Models for Real-Time Situated Reasoning

Giulio Antonio Abbo, Senne Lenaerts, Tony Belpaeme

TL;DR

This work demonstrates real-time, context- and value-aware decision-making for a domestic robot by integrating a multimodal large language model (GPT-4o) with a TurtleBot 4. The system processes vision input through a structured prompting and reasoning pipeline to infer activities, norms, and user preferences, and uses a three-state control loop (observation, cleaning, docking) to decide actions with an explainable trace. Qualitative evaluations in simulated scenes and real-home setups show capabilities to balance cleanliness with user comfort and safety, while acknowledging challenges in real-time performance, bias, and privacy. The study highlights the potential of embodied multimodal reasoning to advance socially aware, adaptable household robotics, and outlines avenues for future work to address biases and efficiency.

Abstract

In this work, we explore how multimodal large language models can support real-time context- and value-aware decision-making. To do so, we combine the GPT-4o language model with a TurtleBot 4 platform simulating a smart vacuum cleaning robot in a home. The model evaluates the environment through vision input and determines whether it is appropriate to initiate cleaning. The system highlights the ability of these models to reason about domestic activities, social norms, and user preferences and take nuanced decisions aligned with the values of the people involved, such as cleanliness, comfort, and safety. We demonstrate the system in a realistic home environment, showing its ability to infer context and values from limited visual input. Our results highlight the promise of multimodal large language models in enhancing robotic autonomy and situational awareness, while also underscoring challenges related to consistency, bias, and real-time performance.

Multimodal Large Language Models for Real-Time Situated Reasoning

TL;DR

This work demonstrates real-time, context- and value-aware decision-making for a domestic robot by integrating a multimodal large language model (GPT-4o) with a TurtleBot 4. The system processes vision input through a structured prompting and reasoning pipeline to infer activities, norms, and user preferences, and uses a three-state control loop (observation, cleaning, docking) to decide actions with an explainable trace. Qualitative evaluations in simulated scenes and real-home setups show capabilities to balance cleanliness with user comfort and safety, while acknowledging challenges in real-time performance, bias, and privacy. The study highlights the potential of embodied multimodal reasoning to advance socially aware, adaptable household robotics, and outlines avenues for future work to address biases and efficiency.

Abstract

In this work, we explore how multimodal large language models can support real-time context- and value-aware decision-making. To do so, we combine the GPT-4o language model with a TurtleBot 4 platform simulating a smart vacuum cleaning robot in a home. The model evaluates the environment through vision input and determines whether it is appropriate to initiate cleaning. The system highlights the ability of these models to reason about domestic activities, social norms, and user preferences and take nuanced decisions aligned with the values of the people involved, such as cleanliness, comfort, and safety. We demonstrate the system in a realistic home environment, showing its ability to infer context and values from limited visual input. Our results highlight the promise of multimodal large language models in enhancing robotic autonomy and situational awareness, while also underscoring challenges related to consistency, bias, and real-time performance.
Paper Structure (4 sections, 1 figure)

This paper contains 4 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Implementation
  3. Observations
  4. Conclusion

Figures (1)

  • Figure 1: The robot deployed in a living room, and an image from the robot's perspective, showing the unusual angle and the partially obstructed view.