Transfer entropy for finite data

TL;DR

The paper introduces a combinatorial, nonparametric reduced transfer entropy that corrects the finite-sample positivity bias and provides automatic statistical interpretation without simulations. By reinterpreting data as finite populations and using a contingency-table encoding, the authors derive $H_C(\mathbf{w}|\mathbf{V})$ and the reduced TE $\mathcal{R}^{(k,l)}_{\mathbf{x}\to \mathbf{y}}$, with a guaranteed non-positive correction $\Delta^{(k,l)}_{\mathbf{x}\to \mathbf{y}}$ and MDL-based model selection. The framework recovers asymptotically standard TE but remains reliable for small $N$ or high cardinality $C$, supports multivariate extensions, and enables automatic lag selection. Through synthetic and real-data experiments, the method yields more robust, sparse, and interpretable information-flow networks compared to conventional TE methods, with practical implications for neuroscience, climate science, finance, and conflict data analysis.

Abstract

Transfer entropy is a widely used measure for quantifying directed information flows in complex systems. While the challenges of estimating transfer entropy for continuous data are well known, it has two major shortcomings for data of finite cardinality: it exhibits a substantial positive bias for sparse bin counts, and it has no clear means to assess statistical significance. By computing information content in finite data streams without explicitly considering symbols as instances of random variables, we derive a transfer entropy measure which is asymptotically equivalent to the standard plug-in estimator but remedies these issues for time series of small size and/or high cardinality, permitting a fully nonparametric assessment of statistical significance without simulation.

Figures (10)

• Figure 1: Transfer entropy of random time series with sparse counts and lags $k=l=1$. Sparsity here is from small $T$, but can also arise from large lags $k,l$ or cardinality $C$. A negative reduced transfer entropy indicates that it is more compressive to transmit $\bm{y}$'s future values $\bm{y}^{(+1)}$ using only its own past values $\bm{Y}^{(-l)}$ than to also include $\bm{x}$'s past values $\bm{X}^{(-k)}$.
• Figure 2: Transfer entropy of synthetic time series. (a) Normalized reduced transfer entropy (Eq. \ref{['eq:normTEreduced']}) versus cross- and auto-correlation noise, for synthetic time series $\bm{x},\bm{y}$ with $\{T,l,C\}=\{100,3,2\}$. The region of statistical significance ($\mathcal{R}>0$) is indicated with a black line. (b) Normalized standard transfer entropy (Eq. \ref{['eq:normTEstandard']}) for the same time series, statistical significance ($p<0.05$ from permutation testing) indicated with a white line. Panels (c) and (d) repeat these experiments for $T=1000$.
• Figure 3: Positivity bias of transfer entropy. (Top) Normalized transfer entropies versus cross-correlation noise, $\beta$, for synthetic time series with $(T,C)=(40,3),(10000,25)$ and $k=l=1$. (Bottom) Standard transfer entropy as a function of $T,C$ for time series $\bm{x},\bm{y}\in \{1,...,C\}^{T}$ generated uniformly at random, with $k=l=1$.
• Figure 4: The transfer entropy correction can substantially impact downstream results. Networks were constructed from hourly sampled time series recording air quality across Hong Kong at 18 different measurement stations (nodes). We place a directed edge $(i,j)$ if the time series $\bm{x}$ at $i$ has a statistically significant transfer entropy with the series $\bm{y}$ at $j$ with $k=l=1\text{ hour}$. The number of edges $E$ in the final network are listed above each panel.
• Figure S1: (Top) Standard and reduced (normalized) transfer entropy versus cross-correlation noise, $\beta$, for synthetically generated time series $\bm{x},\bm{y}$ with $(T,C)=(100,4)$. (Bottom) Same plot for $(T,C)=(1000,10)$. A lag of $k=1$ is imposed between $\bm{x}$ and $\bm{y}$.