Identifying Information-Transfer Nodes in a Recurrent Neural Network Reveals Dynamic Representations

TL;DR

Addresses the challenge of interpreting RNN internal dynamics by identifying information-transfer nodes, termed information relays, using a mutual-information measure across hidden-layer subsets. The method computes $I_R = H(X_{ m in}; X_{ m out}; Y_R \,|\; Y_0)$ to quantify relayed information and uses a greedy algorithm to order nodes by contribution; for recurrent networks, relay information is evaluated at each time point after all inputs are presented to capture possible migration of information over time. The approach is validated on memory and block time-series classification tasks using RNN, GRU, and LSTM architectures, revealing that RNNs exhibit information migration between nodes while GRUs/LSTMs localize information in stable node sets, with concept-specific knockout effects; PCA analyses further show attractor-like latent structure. Overall, the work advances explainability and provides a scalable tool for diagnosing temporal information processing in sequential models.

Abstract

Understanding the internal dynamics of Recurrent Neural Networks (RNNs) is crucial for advancing their interpretability and improving their design. This study introduces an innovative information-theoretic method to identify and analyze information-transfer nodes within RNNs, which we refer to as \textit{information relays}. By quantifying the mutual information between input and output vectors across nodes, our approach pinpoints critical pathways through which information flows during network operations. We apply this methodology to both synthetic and real-world time series classification tasks, employing various RNN architectures, including Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs). Our results reveal distinct patterns of information relay across different architectures, offering insights into how information is processed and maintained over time. Additionally, we conduct node knockout experiments to assess the functional importance of identified nodes, significantly contributing to explainable artificial intelligence by elucidating how specific nodes influence overall network behavior. This study not only enhances our understanding of the complex mechanisms driving RNNs but also provides a valuable tool for designing more robust and interpretable neural networks.

Figures (13)

• Figure 1: Illustration of the block (left) and the memory (right) task.
• Figure 2: Performance of different recurrent neural networks (RNN, GRU, and LSTM) independently trained for two different tasks. At the top row, results for the memory task, and for the block task at the bottom. Networks were trained with either a specific delay of 1, 2, 3, 4, or 5 time points between the information was delivered and the answer obtained (see color legend), or trained with randomly picking on of those 5 delay intervals for each input sequence. Performance was measured for 10 different delays ranging from 0 to 9 - observe that the time delays (for example, 0, 6, 7, 8, or 9) were never observed during training, and thus, performance measured for those delays characterizes the ability of the networks to generalize to the temporal domain. The shadow behind each line shows the standard error within all 20 experiments.
• Figure 3: Visualization of the data the information relay method and the greedy algorithm produces. On top the decrease of information about a hypothetical concept A the set of nodes relays while the set is continuously becoming smaller (black line). The reduction is shown as a plot, and the relay information loss as blue arrows. For simplicity reasons it is assumed here that the order in which nodes are removed goes from 1 to 12, as the later nodes relay the most information. Consequently, the amount of information each node relays can be visualized in the form of a bar at the bottom of the plot. Below that, the same process is shown for a different concept B, the nodes of a network can relay. Here the order of nodes is different, and thus the visualization highlights different nodes.
• Figure 4: Effect of knockouts (setting sets of hidden nodes to $0.0$ for different neural networks (RNN, GRN, and LSTM). The information relay method identified nodes carrying a specific concept (columns A, B, or C, see color). Knockouts were applied to the largest set first and then to progressively smaller sets in the order of importance to the given concept. Accuracy for each concept in shown in their respective colors. Results present averages over all 20 independently trained networks on the memory task. The shadows behind each line are 95% confidence intervals of the mean. The delay used during training was from the interval $[1,5]$.
• Figure 5: Effect of knockouts (setting sets of hidden nodes to $0.0$ for different neural networks (RNN, GRN, and LSTM). Color code and layout is the same as in Figure \ref{['fig:KOmemory']}, except that here the block task was used, and thus the three concepts are direction, size, and brightness.