CMIS-Net: A Cascaded Multi-Scale Individual Standardization Network for Backchannel Agreement Estimation

TL;DR

CMIS-Net addresses backchannel agreement estimation under strong inter-person variability and data imbalance by introducing cascaded multi-scale individual standardization. It employs parallel frame- and sequence-level neutral and emotional encoders, translator modules, and a subtraction-based standardization to extract invariant signals; an implicit data augmentation strategy further enhances generalization. Evaluated on MPIIGroupInteraction, CMIS-Net achieves state-of-the-art performance (e.g., MSE 0.057962), surpassing graph-based and Transformer baselines while revealing modality-specific differences between visual and auditive backchannels. The approach provides a generalizable framework for subtle sequential pattern recognition and may extend to other domains requiring robust cross-person inference.

Abstract

Backchannels are subtle listener responses, such as nods, smiles, or short verbal cues like "yes" or "uh-huh," which convey understanding and agreement in conversations. These signals provide feedback to speakers, improve the smoothness of interaction, and play a crucial role in developing human-like, responsive AI systems. However, the expression of backchannel behaviors is often significantly influenced by individual differences, operating across multiple scales: from instant dynamics such as response intensity (frame-level) to temporal patterns such as frequency and rhythm preferences (sequence-level). This presents a complex pattern recognition problem that contemporary emotion recognition methods have yet to fully address. Particularly, existing individualized methods in emotion recognition often operate at a single scale, overlooking the complementary nature of multi-scale behavioral cues. To address these challenges, we propose a novel Cascaded Multi-Scale Individual Standardization Network (CMIS-Net) that extracts individual-normalized backchannel features by removing person-specific neutral baselines from observed expressions. Operating at both frame and sequence levels, this normalization allows model to focus on relative changes from each person's baseline rather than absolute expression values. Furthermore, we introduce an implicit data augmentation module to address the observed training data distributional bias, improving model generalization. Comprehensive experiments and visualizations demonstrate that CMIS-Net effectively handles individual differences and data imbalance, achieving state-of-the-art performance in backchannel agreement detection.

Figures (7)

• Figure 1: Overview of the CMIS-Net architecture (a), utilizing two cascaded standardization modules: 1. Frame-level standardization (b) extracts instant dynamic features while considering individual-specific variations from each frame; and 2. Sequence-level standardization captures temporal patterns across frames while further disentangling person-specific characteristics. Video inputs are processed through both modules sequentially to obtain generalizable representations for backchannel agreement estimation.
• Figure 2: Training workflow for the translator module and standardization. The neutral feature pipeline (red paths) generates averaged neutral representations from multiple neutral samples, while the emotional feature pipeline (blue paths) performs feature disentanglement through the translator module.
• Figure 3: The Temporal Attention Pooling (TAP) Module.
• Figure 4: The FLTB (left) and FLT (right) Module.
• Figure 5: The ED-LSTM Translator Module.