Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations

Lorenzo Stacchio, Andrea Ubaldi, Alessandro Galdelli, Maurizio Mauri, Emanuele Frontoni, Andrea Gaggioli

TL;DR

Empathic Prompting introduces a modular, prompting-based approach to embed real-time non-verbal affective cues from facial expressions into LLM conversations, conditioning responses without retraining models. The framework uses sensing, mapping, and prompt enrichment to convert valence/arousal and basic emotions into explicit contextual prompts, enabling more empathic and safe language generation in a local deployment. An internal mixed-methods evaluation (N=5) demonstrates consistent integration of affective signals into outputs and improved conversational fluidity, while highlighting design trade-offs such as verbosity and incongruence handling. The work suggests practical impact for domains like healthcare and education, and provides a scalable, auditable blueprint for incorporating additional non-verbal modalities in future research.

Abstract

We present Empathic Prompting, a novel framework for multimodal human-AI interaction that enriches Large Language Model (LLM) conversations with implicit non-verbal context. The system integrates a commercial facial expression recognition service to capture users' emotional cues and embeds them as contextual signals during prompting. Unlike traditional multimodal interfaces, empathic prompting requires no explicit user control; instead, it unobtrusively augments textual input with affective information for conversational and smoothness alignment. The architecture is modular and scalable, allowing integration of additional non-verbal modules. We describe the system design, implemented through a locally deployed DeepSeek instance, and report a preliminary service and usability evaluation (N=5). Results show consistent integration of non-verbal input into coherent LLM outputs, with participants highlighting conversational fluidity. Beyond this proof of concept, empathic prompting points to applications in chatbot-mediated communication, particularly in domains like healthcare or education, where users' emotional signals are critical yet often opaque in verbal exchanges.

Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations

TL;DR

Empathic Prompting introduces a modular, prompting-based approach to embed real-time non-verbal affective cues from facial expressions into LLM conversations, conditioning responses without retraining models. The framework uses sensing, mapping, and prompt enrichment to convert valence/arousal and basic emotions into explicit contextual prompts, enabling more empathic and safe language generation in a local deployment. An internal mixed-methods evaluation (N=5) demonstrates consistent integration of affective signals into outputs and improved conversational fluidity, while highlighting design trade-offs such as verbosity and incongruence handling. The work suggests practical impact for domains like healthcare and education, and provides a scalable, auditable blueprint for incorporating additional non-verbal modalities in future research.

Abstract

We present Empathic Prompting, a novel framework for multimodal human-AI interaction that enriches Large Language Model (LLM) conversations with implicit non-verbal context. The system integrates a commercial facial expression recognition service to capture users' emotional cues and embeds them as contextual signals during prompting. Unlike traditional multimodal interfaces, empathic prompting requires no explicit user control; instead, it unobtrusively augments textual input with affective information for conversational and smoothness alignment. The architecture is modular and scalable, allowing integration of additional non-verbal modules. We describe the system design, implemented through a locally deployed DeepSeek instance, and report a preliminary service and usability evaluation (N=5). Results show consistent integration of non-verbal input into coherent LLM outputs, with participants highlighting conversational fluidity. Beyond this proof of concept, empathic prompting points to applications in chatbot-mediated communication, particularly in domains like healthcare or education, where users' emotional signals are critical yet often opaque in verbal exchanges.
Paper Structure (32 sections, 10 figures, 3 tables)

This paper contains 32 sections, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. System architecture and Implementation
  4. User Chat Web Interface
  5. Middleware and Supervisor Software Implementation
  6. Server Implementation
  7. Use case
  8. Step 1: Onboarding and Visual Priming
  9. Step 2: Initial Interaction and Congruence Handling
  10. Step 3: Navigating Emotional Incongruence
  11. Outcome of the Use Case
  12. Experimental Study and Results
  13. Apparatus
  14. Prompt Design
  15. Large Language Model Performance Evaluation test
  16. ...and 17 more sections

Figures (10)

  • Figure 1: System architecture of the proposed Empathic Prompting Framework.
  • Figure 2: Client UI developed for the Empathic Prompting system. The interface enables natural language interaction between the user and the LLM. In this example (taken from a real conversation with a user, made in Italian, target language for our experimentation), the chatbot responds empathetically to a user’s positive disclosure, explicitly acknowledging emotions and reinforcing conversational alignment.
  • Figure 3: Supervisor-facing control panel of the middleware, allowing connection management, user identification, and start/stop control of Noldus FaceReader data streaming.
  • Figure 4: Prompt template for multimodal analysis of social media post outliers based on engagement.
  • Figure 5: Emphatic System Prompt.
  • ...and 5 more figures