Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nadine: An LLM-driven Intelligent Social Robot with Affective Capabilities and Human-like Memory

Hangyeol Kang, Maher Ben Moussa, Nadia Magnenat-Thalmann

TL;DR

The paper addresses the challenge of endowing social robots with human-like affective capabilities and long-term memory. It proposes SoR-ReAct, an LLM-driven agent embedded in the Nadine platform, which integrates perception, episodic memory via retrieval-augmented generation, affective computing, and tool-enabled reasoning to produce natural, memory-aware behaviors. Key contributions include the SoR-ReAct framework, an LLM-RAG memory system for both static knowledge and dynamic episodic memory, and an affective system that links personality, mood, and emotion to robot behavior, validated through ablation studies. The work advances naturalistic human-robot interaction by enabling persistent user models and emotionally coherent responses, though it currently supports single-user interactions and invites expansion to multi-party settings in future work.

Abstract

In this work, we describe our approach to developing an intelligent and robust social robotic system for the Nadine social robot platform. We achieve this by integrating Large Language Models (LLMs) and skilfully leveraging the powerful reasoning and instruction-following capabilities of these types of models to achieve advanced human-like affective and cognitive capabilities. This approach is novel compared to the current state-of-the-art LLM-based agents which do not implement human-like long-term memory or sophisticated emotional appraisal. The naturalness of social robots, consisting of multiple modules, highly depends on the performance and capabilities of each component of the system and the seamless integration of the components. We built a social robot system that enables generating appropriate behaviours through multimodal input processing, bringing episodic memories accordingly to the recognised user, and simulating the emotional states of the robot induced by the interaction with the human partner. In particular, we introduce an LLM-agent frame for social robots, SoR-ReAct, serving as a core component for the interaction module in our system. This design has brought forth the advancement of social robots and aims to increase the quality of human-robot interaction.

Nadine: An LLM-driven Intelligent Social Robot with Affective Capabilities and Human-like Memory

TL;DR

The paper addresses the challenge of endowing social robots with human-like affective capabilities and long-term memory. It proposes SoR-ReAct, an LLM-driven agent embedded in the Nadine platform, which integrates perception, episodic memory via retrieval-augmented generation, affective computing, and tool-enabled reasoning to produce natural, memory-aware behaviors. Key contributions include the SoR-ReAct framework, an LLM-RAG memory system for both static knowledge and dynamic episodic memory, and an affective system that links personality, mood, and emotion to robot behavior, validated through ablation studies. The work advances naturalistic human-robot interaction by enabling persistent user models and emotionally coherent responses, though it currently supports single-user interactions and invites expansion to multi-party settings in future work.

Abstract

In this work, we describe our approach to developing an intelligent and robust social robotic system for the Nadine social robot platform. We achieve this by integrating Large Language Models (LLMs) and skilfully leveraging the powerful reasoning and instruction-following capabilities of these types of models to achieve advanced human-like affective and cognitive capabilities. This approach is novel compared to the current state-of-the-art LLM-based agents which do not implement human-like long-term memory or sophisticated emotional appraisal. The naturalness of social robots, consisting of multiple modules, highly depends on the performance and capabilities of each component of the system and the seamless integration of the components. We built a social robot system that enables generating appropriate behaviours through multimodal input processing, bringing episodic memories accordingly to the recognised user, and simulating the emotional states of the robot induced by the interaction with the human partner. In particular, we introduce an LLM-agent frame for social robots, SoR-ReAct, serving as a core component for the interaction module in our system. This design has brought forth the advancement of social robots and aims to increase the quality of human-robot interaction.
Paper Structure (17 sections, 10 figures)

This paper contains 17 sections, 10 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Nadine Framework
  4. Overview of the system
  5. Perception Module
  6. Interaction Module
  7. Dialogue management
  8. Affective system
  9. LLM-RAG based memory and knowledge system
  10. LLM Tools
  11. Robot Control Module
  12. Results
  13. Tool use
  14. Affective system
  15. Long-term memory
  16. ...and 2 more sections

Figures (10)

  • Figure 1: Nadine social robot new appearance.
  • Figure 2: Nadine system architecture. The system comprises three key modules: the perception module, the interaction module, and the control module. This study specifically focuses on the interaction module, which equips a social robot with an LLM-powered agent to achieve human-like affective and cognitive capabilities.
  • Figure 3: SoR-ReAct prompt structure
  • Figure 4: SoR-ReAct workflow
  • Figure 5: (a): LLM tools request and response. (b): LLM final response.
  • ...and 5 more figures