Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

TL;DR

The paper addresses grounding large language models (LLMs) in interactive, multimodal robotic perception. It introduces Matcha, a modular framework with an LLM backbone, perception modules (vision, sound, weight, haptics), and a language-conditioned execution policy to perform epistemic actions and continuously reason over environment feedback. Through few-shot prompts and a simulated object-picking task, the work demonstrates that strong LLMs can effectively plan and justify actions when multimodal sensory signals are translated into natural language. The findings show substantial gains from multimodal grounding, while also highlighting limitations with indistinct sensory descriptions and weaker LLMs, guiding future directions in grounding, prompt design, and multimodal LLM development.

Abstract

Programming robot behavior in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. Matcha (Multimodal environment chatting) agent, an interactive perception framework, is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability. The project website can be found at https://matcha-agent.github.io.

Figures (5)

• Figure 1: Given instruction from a human, the robot recurrently "chats" with the environment to obtain sufficient information for achieving the task. An LLM generates action commands to interactively perceive the environment and, in response, the environment provides multimodal feedback (MF) through multimodal perception modules.
• Figure 2: Overview of Matcha. The framework contains an LLM, multimodal perception modules, and a language-conditioned control policy. These components communicate with each other with natural language as the intermediate representation. Three types of language information are involved in composing the prompt: I is a language instruction from the user, C is a language command produced by the LLM, and F is semantic feedback from multimodal perceptions. Dotted lines indicate possibly evoking paths.
• Figure 3: A successful example in which the robot deduces "fiber" material with indistinct descriptions of impact sound.
• Figure 4: A successful example with a distinct description of impact sound. This example shows that by leveraging multimodal perception, LLM rectifies the misclassification that may occur when relying solely on sound modules.
• Figure 5: An example in which the agent fails to distinguish glass and ceramic in the setup of using indistinct descriptions of impact sound.