Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multimodal Framework for Human-Multi-Agent Interaction

Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal

Abstract

Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.

A Multimodal Framework for Human-Multi-Agent Interaction

Abstract

Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
Paper Structure (10 sections, 3 figures)

This paper contains 10 sections, 3 figures.

Table of Contents

  1. Introduction
  2. System Design
  3. Agent Module
  4. Perception
  5. Planning
  6. Action
  7. Multi-Agent Interaction
  8. System Demonstration and Results
  9. Discussion
  10. Conclusion

Figures (3)

  • Figure 1: Multimodal multi-agent human–robot interaction scenario. A single human user engages simultaneously with two humanoid robot agents through speech and non-verbal cues.
  • Figure 2: Developed framework overview (a) Internal closed-loop module of a single robot agent, showing multimodal perception, LLM-based planning, and embodied action execution via an action library. (b) Multi-agent interaction setup, where multiple agents (e.g., Sam and Journey) interact with a human user under centralized coordination.
  • Figure 3: A representative interaction run from our system demonstration. A human user engages two humanoid robots in shared space through speech and visual cues. The central coordinator selects the most contextually relevant agent(s) to respond, enabling sequential, non-overlapping dialogue and grounded physical actions such as verbal replies and locomotion.