Interpretability of the Intent Detection Problem: A New Approach

TL;DR

The paper investigates the interpretability of RNN-based intent detection through a dynamical-systems lens, treating sentences as trajectories in hidden state space. It shows that on the balanced SNIPS dataset the network forms a low-dimensional manifold with distinct intent clusters and trajectories that move toward region-specific endpoints aligned with readout vectors; when tested on the imbalanced ATIS dataset, the geometry distorts for low-frequency intents. A two-part diagnostic framework (Geometric Separation and Readout Alignment) identifies four mechanistic patterns of success and failure, linking performance to state-space geometry and readout alignment. The findings offer mechanistic, geometric explanations for real-world performance disparities and suggest extensions to out-of-domain detection and transformer architectures, with practical impact on interpretability and robustness in conversational AI.

Abstract

Intent detection, a fundamental text classification task, aims to identify and label the semantics of user queries, playing a vital role in numerous business applications. Despite the dominance of deep learning techniques in this field, the internal mechanisms enabling Recurrent Neural Networks (RNNs) to solve intent detection tasks are poorly understood. In this work, we apply dynamical systems theory to analyze how RNN architectures address this problem, using both the balanced SNIPS and the imbalanced ATIS datasets. By interpreting sentences as trajectories in the hidden state space, we first show that on the balanced SNIPS dataset, the network learns an ideal solution: the state space, constrained to a low-dimensional manifold, is partitioned into distinct clusters corresponding to each intent. The application of this framework to the imbalanced ATIS dataset then reveals how this ideal geometric solution is distorted by class imbalance, causing the clusters for low-frequency intents to degrade. Our framework decouples geometric separation from readout alignment, providing a novel, mechanistic explanation for real world performance disparities. These findings provide new insights into RNN dynamics, offering a geometric interpretation of how dataset properties directly shape a network's computational solution.

Figures (9)

• Figure 1: (a) Folded representation of a Recurrent Neural Network (RNN), highlighting the recurrent connection within the architecture. (b) Unfolded representation of an RNN, explicitly showing the flow of time. At each time step $t$, the RNN processes an input token $\mathbf{x}_t$ (e.g. words in a sentence like "add", "movie", or "playlist"), updates its hidden state $\mathbf{h}_t$ based on the previous state $\mathbf{h}_{t-1}$, and generates an output $\mathbf{y}_t$. The initial hidden state $\mathbf{h}_0$ represents the starting point of the RNN's before processing any input tokens.
• Figure 2: Sequence of hidden states $\mathbf{h}_1, \dots, \mathbf{h}_T$ associated with a tokenized input sentence $\mathbf{x}_1, \dots, \mathbf{x}_T$ (e.g. "add ... movie ... playlist") as it is processed by an RNN. The hidden states $\mathbf{h}_t$ evolve in response to input tokens $\mathbf{x}_t$, representing the progression of the network's internal dynamics. The initial hidden state $\mathbf{h}_0$ serves as the starting point before any tokens are processed. The final hidden state $\mathbf{h}_T$ captures the cumulative information of the input sequence and is used to generate the prediction $\mathbf{y}_T$ via the readout layer. The type of recurrent cell (Vanilla, GRU, LSTM) influences how the network captures sequential dependencies.
• Figure 3: Variance explained by principal components for the hidden states of RNNs trained on the SNIPS dataset. (a) PCA analysis for a GRU-based RNN with embedding size = 10 and hidden layer size = 20. The bars represent the variance explained by individual components, while the green curve shows the cumulative variance. The intrinsic dimensionality (id = 4) is marked where the cumulative variance surpasses the 95% threshold (dashed horizontal line). (b) Comparison of intrinsic dimensionalities for GRU, LSTM, and vanilla RNNs with embedding size = 10 and hidden size = 20. The cumulative variance curves show that GRU and vanilla RNNs reach $id = 4$, while LSTM requires $id = 5$ to exceed the threshold.
• Figure 4: State space dimensionality and accuracy of RNNs trained on the SNIPS dataset for different combinations of embedding and hidden layer size. The top row shows the intrinsic dimensionality (id) of the state space for (a) GRU RNNs, (b) LSTM RNNs and (c) Vanilla RNNs. The bottom row displays the corresponding classification accuracy for the same architectures: (d) GRU, (e) LSTM, and (f) Vanilla RNNs.
• Figure 5: Top-2 and top-3 PCA projections of the state space of a GRU(emb:16,hid:16) trained on the SNIPS dataset. The hidden states are colored based on the intent label of its corresponding sentence. (a) 3D PCA projection. (b) 2D PCA projection, highligting intent clusters. (c) Silhouette score from a K-means clustering analysis of the state space, showing a score of 0.57 indicating a moderate level of cluster separation.