Adaptive Speech Emotion Representation Learning Based On Dynamic Graph

TL;DR

Addressing the limitation of static graphs in speech emotion recognition, the paper proposes an adaptive graph representation learning framework that builds dynamic graphs on sliding-window subsequences and updates node representations with a degree-aware Dice-based adjacency. It defines a Dice-inspired similarity matrix $A_{Dice}^s$ and a segment-specific GCN with $A^s = \tilde{A}^s + \varphi A_{Dice}^s + I$ and $H^s = \sigma(\hat{D}_s^{-\frac{1}{2}} A^s \hat{D}_s^{-\frac{1}{2}} X^s W^s)$, along with a learnable adjacency $A_{Learn}$ and a joint loss $\mathcal{L} = \mathcal{L}_{GC} + \mathcal{L}_{GL}$ to optimize graph structure and prediction. Empirical results on IEMOCAP and RAVDESS show improvements over state-of-the-art graph-based and conventional models, underscoring the benefit of capturing local and global context in speech. The work advances SER by enabling dynamic graph evolution and degree-aware edge weighting, with potential for multimodal sequential data.

Abstract

Graph representation learning has become a hot research topic due to its powerful nonlinear fitting capability in extracting representative node embeddings. However, for sequential data such as speech signals, most traditional methods merely focus on the static graph created within a sequence, and largely overlook the intrinsic evolving patterns of these data. This may reduce the efficiency of graph representation learning for sequential data. For this reason, we propose an adaptive graph representation learning method based on dynamically evolved graphs, which are consecutively constructed on a series of subsequences segmented by a sliding window. In doing this, it is better to capture local and global context information within a long sequence. Moreover, we introduce a weighted approach to update the node representation rather than the conventional average one, where the weights are calculated by a novel matrix computation based on the degree of neighboring nodes. Finally, we construct a learnable graph convolutional layer that combines the graph structure loss and classification loss to optimize the graph structure. To verify the effectiveness of the proposed method, we conducted experiments for speech emotion recognition on the IEMOCAP and RAVDESS datasets. Experimental results show that the proposed method outperforms the latest (non-)graph-based models.

Figures (3)

• Figure 1: Overview of our proposed model. The feature matrix $\mathbf{X}^{s}$, the graph structure (i.e., adjacency matrix $\mathbf{A}^{s}$), and the weight matrix $\mathbf{W}^{s}$ of the graph are dynamically changed with different audio segments $s$. Among them, the graph structure is mainly obtained by our proposed matrix calculation method ($\mathbf{A}_{Dice}^{s}$), and combined with the graph structure loss and classification loss to jointly optimize the graph structure.
• Figure 2: Graph Construction. Given an audio segment $s$, frame it in 25ms steps to obtain $m$ frames. Then follow the frame-to-node transformation to get $m$ nodes. The list of edges is determined by connecting neighboring nodes and randomly connecting distant nodes. The relationship between nodes is jointly determined by the proposed matrix computation method and learning during the training. Each node has a node feature vector $x_{i}$ associated with it.
• Figure 3: Comparison of traditional Dice similarity $\left ( \mathbf{S}_{Dice} \right )$ and proposed Dice similarity $\left ( \mathbf{A}_{Dice} \right )$. $v_{2}$ and $v_{3}$ have the same number of neighbors regarding the target node $v_{1}$; whilst the node with a larger degree ($v_{3}$) has a higher influence.