DEEMO: De-identity Multimodal Emotion Recognition and Reasoning

TL;DR

DEEMO tackles privacy-preserving emotion understanding by introducing de-identified video, audio, and transcription cues along with non-facial body language (NFBL). It provides the DEEMO dataset with two subsets (DEEMO-NFBL for NFBL annotations and DEEMO-MER for emotion recognition plus reasoning) and the DEEMO-LLaMA model, a multimodal LLM trained to perform both recognition and reasoning under identity-free constraints. Experimental results show state-of-the-art performance on de-identity emotion recognition (74.49% accuracy, 74.45% F1) and emotion reasoning (6.20 clue overlap, 7.66 label overlap), confirming the effectiveness of de-identified, NFBL-focused signals. This work advances privacy-preserving affective computing and ethical AI by demonstrating robust emotion understanding without identity leakage, enabling responsible real-world applications.

Abstract

Emotion understanding is a critical yet challenging task. Most existing approaches rely heavily on identity-sensitive information, such as facial expressions and speech, which raises concerns about personal privacy. To address this, we introduce the De-identity Multimodal Emotion Recognition and Reasoning (DEEMO), a novel task designed to enable emotion understanding using de-identified video and audio inputs. The DEEMO dataset consists of two subsets: DEEMO-NFBL, which includes rich annotations of Non-Facial Body Language (NFBL), and DEEMO-MER, an instruction dataset for Multimodal Emotion Recognition and Reasoning using identity-free cues. This design supports emotion understanding without compromising identity privacy. In addition, we propose DEEMO-LLaMA, a Multimodal Large Language Model (MLLM) that integrates de-identified audio, video, and textual information to enhance both emotion recognition and reasoning. Extensive experiments show that DEEMO-LLaMA achieves state-of-the-art performance on both tasks, outperforming existing MLLMs by a significant margin, achieving 74.49% accuracy and 74.45% F1-score in de-identity emotion recognition, and 6.20 clue overlap and 7.66 label overlap in de-identity emotion reasoning. Our work contributes to ethical AI by advancing privacy-preserving emotion understanding and promoting responsible affective computing.

Figures (5)

• Figure 1: Previous multimodal emotion understanding tends to rely on identity-sensitive information (e.g., facial expression). In contrast, DEEMO leverages de-identified video and audio, identity-free audio transcriptions, and non-facial body language (NFBL) cues to enable privacy-preserving emotion recognition and reasoning.
• Figure 2: Exapmles of DEEMO-NFBL.
• Figure 3: Statistical visualization of the DEEMO dataset. The bar chart displays the frequency of 37 defined NFBL categories, with colors indicating their types: self-manipulations (yellow), self-protection behaviors (green), and manipulation of objects (blue). The three pie charts provide dataset-level statistics, including gender distribution, emotion labels of DEEMO-MER, and three types of NFBL distributions of DEEMO-NFBL. Best viewed digitally in color and zoomed in.
• Figure 4: Overview of the annotation workflow for the DEEMO-MER. The process consists of three stages: 1) data pre-labeling via multimodal de-identified cues (video, audio, transcription, and NFBL); 2) emotion recognition annotation through LLM-based pre-labeling followed by human review; 3) multimodal emotion reasoning annotation with LLM and human review. All LLM-generated annotations are cross-reviewed by two trained annotators to ensure labeling quality and consistency.
• Figure 5: Overview of the proposed DEEMO-LLaMA framework. The model takes de-identified video and audio as input. Each modality is processed by a dedicated encoder followed by a Q-former and a linear projection layer, and the resulting embeddings are integrated into a unified prompt for input to an LLM. In the emotion reasoning answer, different colors are used to highlight how DEEMO-LLaMA leverages various modality-specific cues (visual cues, acoustic cues, audio transcription cues and non facial body language cues) to infer emotional states.