Which bits went where? Past and future transfer entropy decomposition with the information bottleneck

Abstract

Whether the system under study is a shoal of fish, a collection of neurons, or a set of interacting atmospheric and oceanic processes, transfer entropy measures the flow of information between time series and can detect possible causal relationships. Much like mutual information, transfer entropy is generally reported as a single value summarizing an amount of shared variation, yet a more fine-grained accounting might illuminate much about the processes under study. Here we propose to decompose transfer entropy and localize the bits of variation on both sides of information flow: that of the originating process's past and that of the receiving process's future. We employ the information bottleneck (IB) to compress the time series and identify the transferred entropy. We apply our method to decompose the transfer entropy in several synthetic recurrent processes and an experimental mouse dataset of concurrent behavioral and neural activity. Our approach highlights the nuanced dynamics within information flow, laying a foundation for future explorations into the intricate interplay of temporal processes in complex systems.

Figures (3)

• Figure 1: Localizing transfer entropy in the past and future.(a) If the past of a large red fish (the source) helps predict the future of a small blue fish (the target) after accounting for its past, then there is positive transfer entropy. Equivalently, if the future of the blue fish helps infer the past of the red fish after accounting for the blue fish's past, then there is positive transfer entropy. (b) A machine learning scheme to extract the transfer entropy from the source's past, using the information bottleneck (IB). (c) An analogous scheme to extract the transfer entropy from the target's future.
• Figure 2: Transferred entropy in binary-valued recurrent networks.(a)Left: The update rule for four processes, where nodes without inputs (blue, orange) are randomly sampled at each timestep. The source and target processes are indicated by the shaded boxes and marked $X$ and $Y$, respectively. Middle: Distributed information planes that visualize the decomposition of transfer entropy in the source's past and the target's future. Right: The share of transfer entropy residing in different timesteps of the source's past (top) and target's future (bottom) (taken from the rightmost point of the trajectories in the middle). (b) Same as panel a, but with different target processes. (c) Same as panel a, but with randomly generated connection weights and an integrate-and-fire scheme.
• Figure 3: Transfer entropy between brain and behavior.(a) Concurrent neural and behavioral recordings were taken of six mice; example time series shown on the right with the brain regions shown with matching colors in the atlas. (b) Pairwise transfer entropy between the 23 brain regions, the reference average signal (Avg), and three behavioral streams. (c) Transfer entropy decomposition from behaviors to the purple region from a, the primary somatosensory area for the nose (SSp-n). The instantaneous Kullback-Leibler (KL) cost in natural units (nats) per channel (black) is shown concurrently with the raw time series (colored).