Beyond Discrete Categories: Multi-Task Valence-Arousal Modeling for Pet Vocalization Analysis

TL;DR

This work advances pet emotion recognition by shifting from discrete categories to continuous Valence-Arousal (VA) representations. It introduces an automatic VA labeling pipeline to scale to over 42k pet vocalizations and develops a transformer-based multi-task framework that jointly predicts Valence, Arousal, and auxiliary attributes (discrete emotion, size, gender). The approach yields strong Valence correlations ($r=0.9024$) and competitive Arousal ($r=0.7155$), while ablation studies demonstrate substantial gains from auxiliary tasks. The method supports real-time edge deployment and cross-breed generalization, with broad implications for welfare monitoring, veterinary diagnostics, and consumer pet-care devices. Limitations include labeling biases, Arousal estimation simplicity, species diversity, and lack of temporal dynamics, pointing to future work in human validation, richer temporal features, and real-world deployment validation.

Abstract

Traditional pet emotion recognition from vocalizations, based on discrete classification, struggles with ambiguity and capturing intensity variations. We propose a continuous Valence-Arousal (VA) model that represents emotions in a two-dimensional space. Our method uses an automatic VA label generation algorithm, enabling large-scale annotation of 42,553 pet vocalization samples. A multi-task learning framework jointly trains VA regression with auxiliary tasks (emotion, body size, gender) to enhance prediction by improving feature learning. Our Audio Transformer model achieves a validation Valence Pearson correlation of r = 0.9024 and an Arousal r = 0.7155, effectively resolving confusion between discrete categories like "territorial" and "happy." This work introduces the first continuous VA framework for pet vocalization analysis, offering a more expressive representation for human-pet interaction, veterinary diagnostics, and behavioral training. The approach shows strong potential for deployment in consumer products like AI pet emotion translators.

Figures (5)

• Figure 1: Multi-task VA emotion model architecture. The Audio Transformer encoder (6 layers, 512 hidden dimensions, 8 attention heads) processes Mel spectrograms and feeds into 5 task-specific heads: Valence regression (Tanh activation, range [-1,1]), Arousal regression (Sigmoid, [0,1]), Emotion classification (8 classes), Size classification (3 classes), and Gender classification (2 classes). The multi-task loss combines MSE for VA regression and cross-entropy for classification tasks.
• Figure 2: Training curves over 40 epochs. (a) Validation VA MAE decreases from 0.2560 to 0.1124 (-56.1%). (b) Valence Pearson correlation increases from 0.3282 to 0.9024. (c) Arousal Pearson correlation improves from 0.4110 to 0.7155. Red stars mark the best performance for each metric.
• Figure 3: Predicted versus ground-truth VA values on the validation set. (a) Valence predictions show strong correlation ($r = 0.9024$) with tight clustering around the diagonal. (b) Arousal predictions exhibit moderate correlation ($r = 0.7155$) with more dispersion in the mid-range. Red dashed lines indicate perfect prediction.
• Figure 4: VA space visualization colored by emotion labels. Eight discrete emotions form distinct clusters in the continuous VA space. Negative emotions (fearful, anxious, separation anxiety) occupy the left side (low Valence), while positive emotions (excited, playful, content) cluster on the right (high Valence). Arousal separates calm emotions (content, bottom) from activated emotions (fearful, excited, top). Dashed lines indicate VA space quadrants.
• Figure 5: Ablation study training curves comparing validation VA MAE across different configurations. The Full MTL configuration (bottom curve) achieves the fastest convergence and lowest final VA MAE. All auxiliary task configurations significantly outperform the baseline.