Simple physical systems as a reference for multivariate information dynamics

TL;DR

This work investigates how information-theoretic measures—mutual information, transfer entropy, PID, and integrated information—relate to the mechanistic structure of a simple Gaussian random-walker model with nearest-neighbor coupling. Using an Ornstein-Uhlenbeck–type framework, the authors derive analytical expressions and perform exact calculations for small systems to reveal how coupling strength $\gamma$ and timescale influence information flow at microscopic versus macroscopic levels. They show that information measures can align with underlying mechanics when focusing on microscopic components, short timescales, and causal perturbations, but can mislead when coarse-grained variables (like the center of mass) are used or when stationary versus dynamic contributions are entangled. The study also highlights phenomena such as emergence and autonomy, arguing that certain information- theoretic indicators may reflect statistical autonomy rather than genuine higher-order mechanistic coupling. Overall, causal interventions and scale-aware analyses are proposed as crucial for interpreting information dynamics in real-world complex systems.

Abstract

Understanding a complex system entails capturing the non-trivial collective phenomena that arise from interactions between its different parts. Information theory is a flexible and robust framework to study such behaviours, with several measures designed to quantify and characterise the interdependencies among the system's components. However, since these estimators rely on the statistical distributions of observed quantities, it is crucial to examine the relationships between information-theoretic measures and the system's underlying mechanistic structure. To this end, here we present an information-theoretic analytical investigation of an elementary system of interactive random walkers subject to Gaussian noise. Focusing on partial information decomposition, causal emergence, and integrated information, our results help us develop some intuitions on their relationship with the physical parameters of the system. For instance, we observe that uncoupled systems can exhibit emergent properties, in a way that we suggest may be better described as ''statistically autonomous''. Overall, we observe that in this simple scenario information measures align more reliably with the system's mechanistic properties when calculated at the level of microscopic components, rather than their coarse-grained counterparts, and over timescales comparable with the system's intrinsic dynamics. Moreover, we show that approaches that separate the contributions of the system's dynamics and steady-state distribution (e.g. via causal perturbations) may help strengthen the interpretation of information-theoretic analyses.

Figures (18)

• Figure 1: Mutual information changes with both initial conditions (input) and temporal evolution (channel). Mutual information $I(f(0);f(t))$ between the signal at time $t=0$ and time $t=1$ for different values of $(\alpha,\lambda)$. In the left panel, the swapping rate is fixed at $\lambda=0.1$, providing a non-monotonic MI as $\alpha$ varies. In the right panel, for fixed $\alpha=0.5$, the mutual information decreases monotonously as the swapping rate $\lambda$ induces more randomness in the system. See App. \ref{['app:swap_sig']} for details.
• Figure 2: Transfer entropy between random walkers increases with $\gamma$ as $\mathcal{O}(\gamma^2)$ for $N=2$ RWs. For the system of $N=2$ RWs, the transfer entropy between the two random walkers $\mathcal{T}(\Delta_i(t); \Delta_j(t^{\prime}))$ increases with the system's interaction strength (App. \ref{['app:2-RWcase_TE']}). This also corresponds to the TE between c.o.m. and a random walker $\mathcal{T}(V(t); \Delta_j(t^{\prime}))$. Colours represent different values of $\gamma$, and $t^{\prime}\equiv t+1$.
• Figure 3: Interaction strength increases correlations, thus reducing synergy in small systems. Partial Information Decomposition on the system of $N=2$ random walkers (App. \ref{['app:2-RWcase']}). (a) PID with $X=\Delta_j(t)$, $Y=\Delta_i(t)$, $Z=V(t^{\prime})$, $i\ne j$ (case (1)): redundancy correlates with the interaction strength, while synergy anticorrelates. (b) PID with ${X=\Delta_j(t)}$, ${Y=V(t)}$, $Z=\{\Delta_j(t^{\prime}), V(t^{\prime})\}$, $i\ne j$ (case (2)), which also corresponds to the PID with ${X=\Delta_j(t)}$, ${Y=\Delta_i(t)}$, $Z=\{\Delta_j(t^{\prime}), \Delta_i(t^{\prime})\}$, $i\ne j$ (case (3)): TDMI, synergy, and redundancy anticorrelate with the interaction strength. The dashed lines correspond to the joint mutual information, the dash-dotted lines to redundancy, and the solid lines to synergy. Colours represent different values of $\gamma$, while the black line is independent of $\gamma$. Results are shown for $t=2$.
• Figure 4: Causal intervention can disentangle dynamical and steady-state informational contributions: normalised synergy increases with $\gamma$ in the perturbed system. Normalised synergy for the PID with ${X=\Delta_j(t)}$, ${Y=\Delta_i(t)}$, $Z=\{\Delta_j(t^{\prime}), \Delta_i(t^{\prime})\}$, $i\ne j$ on the system of $N=2$ random walkers (a) decreases in the original system (App. \ref{['app:2-RWcase']}), and (b) increases in the causally perturbed system (App. \ref{['app:2-RWcase_intervention']}). Results are shown for $t=2$.
• Figure 5: Integrated information between random walkers correlates with interaction strength at short timescales . (a) Both whole-minus-sum ($\Phi_{\mathrm{WMS}}$) and revised ($\Phi_{\mathrm{R}}$) integrated information between random walkers correlate with interaction strength. (b) However, when the centre of mass is added to the calculation ($\Phi_{\mathrm{WMS}}^V,\Phi_{\mathrm{R}}^V$), this relationship is reversed. Results are shown for $t=2$.