I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue

TL;DR

The paper tackles the limitation of passive SDSs by proposing an anticipatory framework that predicts a user’s future affective reaction in the next turn for both speech and laughter. It formalizes the approach as a three-component loop—emotion prediction, emotion recognition, and self-correction—and uses a predictor $EM_{prd} = LR(EM_{cur}, DA_{cur})$ to drive proactive planning. Preliminary analyses in human-robot dialogue show strong arousal mimicry with $PCC \approx 0.78$ and that valence depends on dialogue acts, supporting feasibility; the laughter pathway demonstrates a workable mapping from system to user laughter to forecast future laughter events. These findings suggest that anticipatory SDSs can enable proactive, affective interactions with applications in healthcare and education, with future work focusing on incorporating dialogue history and conducting user studies.

Abstract

Current Spoken Dialogue Systems (SDSs) often serve as passive listeners that respond only after receiving user speech. To achieve human-like dialogue, we propose a novel future prediction architecture that allows an SDS to anticipate future affective reactions based on its current behaviors before the user speaks. In this work, we investigate two scenarios: speech and laughter. In speech, we propose to predict the user's future emotion based on its temporal relationship with the system's current emotion and its causal relationship with the system's current Dialogue Act (DA). In laughter, we propose to predict the occurrence and type of the user's laughter using the system's laughter behaviors in the current turn. Preliminary analysis of human-robot dialogue demonstrated synchronicity in the emotions and laughter displayed by the human and robot, as well as DA-emotion causality in their dialogue. This verifies that our architecture can contribute to the development of an anticipatory SDS.

Figures (2)

• Figure 1: The proposed architecture for anticipatory dialogue.
• Figure 2: Valence of the robot's and human's speech during an example dialogue session shows a mimicry pattern.