Enhancing Apparent Personality Trait Analysis with Cross-Modal Embeddings

TL;DR

The paper tackles automatic prediction of Big Five apparent personality traits from short multimodal video clips, addressing the challenge of underrepresented extreme values. It introduces a cross-modal embedding framework with a Siamese network and extends the Multi-Similarity loss to handle all five traits, emphasizing extreme samples through a modifiedhard-example mining strategy. A multi-stage training scheme combines modality-specific networks with cross-modal embeddings and fusion steps, achieving a notable improvement of $0.0033$ in MAE on ChaLearn First Impressions V2 over a strong baseline and enhancing extreme-case predictions. The approach advances robust, multimodal personality assessment with potential applications in human-machine interaction, clinical research, and surveillance, while outlining future refinements like end-to-end training and richer feature representations.

Abstract

Automatic personality trait assessment is essential for high-quality human-machine interactions. Systems capable of human behavior analysis could be used for self-driving cars, medical research, and surveillance, among many others. We present a multimodal deep neural network with a Siamese extension for apparent personality trait prediction trained on short video recordings and exploiting modality invariant embeddings. Acoustic, visual, and textual information are utilized to reach high-performance solutions in this task. Due to the highly centralized target distribution of the analyzed dataset, the changes in the third digit are relevant. Our proposed method addresses the challenge of under-represented extreme values, achieves 0.0033 MAE average improvement, and shows a clear advantage over the baseline multimodal DNN without the introduced module.

Figures (4)

• Figure 1: Pipeline of the proposed method for enhanced Big Five personality trait prediction. Visual, acoustic, and textual information are processed with modality-specific subnetworks. The hidden representations are projected into a shared embedding space with a Siamese network to exploit mutual information of different information sources implicitly. The shared embedding space of the 128D auxiliary vectors is illustrated by colored circles in 2D. The extracted multimodal hidden representations and the cross-modal embeddings are fused before the final Big Five prediction. The training procedure consists of multiple learning stages (LS). FC: fully-connected, Bi-GRU: bidirectional gated recurrent unit, $\oplus$: concatenation operator. The numbers within blocks indicate the number of hidden units used. Multiple values imply stacked layers.
• Figure 2: Examples of the First Impression V2 dataset. For each video the ground truth Big Five scores are provided. For each trait, the first two samples instantiate the high extremes, and the last two examples demonstrate the low extremes of a given trait.
• Figure 3: Personality trait class definitions. Continuous ground truth values are segmented into 4 classes. The thresholds are determined using the mean and standard deviation calculated on the train set trait-wise. Samples from C1 and C4 are the low extremes and high extremes, respectively.
• Figure 4: Visualization of 2-component PCA of cross- and multimodal embeddings of the "test" set (a), showing NEUroticism ground truth values and class labels. The audio, video and text modalities are drawn with circle, square and cross, respectively. The four personality classes are represented with colors, where the blue is the low extreme (C1), and the red is the high extreme class (C4). In the (b) and (c), we emphasize embeddings within the two extreme poles of NEUroticism.