HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs

TL;DR

HARMONI tackles the challenge of sustained, multi-user personalization in socially assistive robotics by coupling four modules—Perception, World Modeling, User Modeling, and Generation—with a privacy-aware LLM-based pipeline. It introduces online memory management across long-term user profiles and short-term conversational context, validated through scenario-driven nursing-home experiments and a hospital study, achieving ~90% speaker identification and ~98% user profile retrieval, with notable gains in personalization quality. The framework demonstrates robust multi-user grounding, low latency, and high usability (SUS ≈ 82.4), highlighting practical impact for elder care and sensitive environments. By integrating multimodal perception, memory-based reasoning, and ethical safeguards, HARMONI advances toward real-world, socially intelligent robotic systems that adapt to individual and group needs while respecting privacy and safety constraints.

Abstract

Existing human-robot interaction systems often lack mechanisms for sustained personalization and dynamic adaptation in multi-user environments, limiting their effectiveness in real-world deployments. We present HARMONI, a multimodal personalization framework that leverages large language models to enable socially assistive robots to manage long-term multi-user interactions. The framework integrates four key modules: (i) a perception module that identifies active speakers and extracts multimodal input; (ii) a world modeling module that maintains representations of the environment and short-term conversational context; (iii) a user modeling module that updates long-term speaker-specific profiles; and (iv) a generation module that produces contextually grounded and ethically informed responses. Through extensive evaluation and ablation studies on four datasets, as well as a real-world scenario-driven user-study in a nursing home environment, we demonstrate that HARMONI supports robust speaker identification, online memory updating, and ethically aligned personalization, outperforming baseline LLM-driven approaches in user modeling accuracy, personalization quality, and user satisfaction.

Figures (8)

• Figure 1: Overview of HARMONI. The proposed framework consists of four modules: (i) a multimodal perception module that identifies the active speaker and extracts the query from visual and auditory inputs; (ii) a world modeling module that maintains representations of all users and the ongoing conversation within a session; (iii) a user modeling module that retrieves and updates speaker-specific profiles for long-term personalization; and (iv) a generation module that produces contextually grounded and personalized responses conditioned on the speaker, their profile, surrounding users, and the conversational context.
• Figure 2: Overview of a challenging multi-turn interaction scenario. The robot dynamically switches context to a new user (Resident B) upon interruption and accurately resolves anaphora (e.g., "the appointment") in follow-up queries by maintaining the updated conversational state, clearly distinguishing it from the initial interaction with Resident A.
• Figure 3: Overview of the proposed user interface architecture. The interface integrates (i) Perception (green components), which identifies active speakers and transcribes input; (ii) User Modeling (blue components), which retrieves and updates user profiles with long-term memory; (iii) and Generation (red components), which produces contextually grounded and personalized responses. The World Modeling, which maintains short-term conversational context and environmental state, is stored at the user interface for simplicity. The interface transparently displays updated user profiles and dialogue history, supporting explainability and real-time adaptation in multi-user scenarios.
• Figure 4: Example of speaker detection within an ego-centric video. (a) Input video frame from egocentric perspective. (b) Perception module output with face grid indicating detected speakers.
• Figure 5: Comparison of models on reply quality (Q3) and latency under different personalization settings (Q4) for PersonaFeedbackpersonafeedback. Scores (a,b) are obtained using an LLM-as-a-Judge, evaluating both answer quality and personalization. Latency (c,d) measures response time. We compare three configurations: direct inference with the base model (no profile), inference with the full user profile, and our method, which retrieves only pertinent profile features via similarity with the query.