Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics

TL;DR

The paper derives general upper bounds on the rates of time-dependent f-divergences (Tsallis, Renyi, KL) between evolving distributions, using temporal Fisher information to quantify the speed of stochastic dynamics. It presents two bound families: a universal, purely kinematic set (B1–B2) applicable to any dynamics, and a Markov-specific, thermodynamic-kinematic set (B3–B6) that incorporates entropy production and activity. The bounds translate into bounds on the rate of mutual information and offer insights for estimating minimal dissipation and speed in nonequilibrium systems, with applications ranging from nonequilibrium thermodynamics to neuroscience and learning. Overall, the results connect information gains, predictability, and thermodynamic costs in complex networks, providing practical tools for bounding dynamical speeds and information-processing capabilities under dissipation constraints.

Abstract

Statistical divergences are important tools in data analysis, information theory, and statistical physics, and there exist well known inequalities on their bounds. However, in many circumstances involving temporal evolution, one needs limitations on the rates of such quantities, instead. Here, several general upper bounds on the rates of some f-divergences are derived, valid for any type of stochastic dynamics (both Markovian and non-Markovian), in terms of information-like and/or thermodynamic observables. As special cases, the analytical bounds on the rate of mutual information are obtained. The major role in all those limitations is played by temporal Fisher information, characterizing the speed of global system dynamics, and some of them contain entropy production, suggesting a link with stochastic thermodynamics. Indeed, the derived inequalities can be used for estimation of minimal dissipation and global speed in thermodynamic stochastic systems. Specific applications of these inequalities in physics and neuroscience are given, which include the bounds on the rates of free energy and work in nonequilibrium systems, limits on the speed of information gain in learning synapses, as well as the bounds on the speed of predictive inference and learning rate. Overall, the derived bounds can be applied to any complex network of interacting elements, where predictability and thermodynamics of network dynamics are of prime concern.

Figures (2)

• Figure 1: Rate of $\alpha$-coefficient in comparison to its various upper bounds as functions of time for $\alpha=2$, corresponding to Pearson divergence. Solid line (blue) corresponds to exact value of $|dC_{2}/dt|$ and was computed from Eq. (6). Upper bounds on $|dC_{2}/dt|$, i.e., B1-B6 are shown as: dashed (red) line for B1; dotted (black) line for B2; crosses (purple) for B3; diamonds (green) for B4; circles (light blue) for B5; and pluses (yellow) for B6. Note that the best estimates for $|dC_{2}/dt|$ are given by the kinematic bounds B1 (dashed line) and B2 (dotted line), but the former is closer to the actual value of $|dC_{2}/dt|$. The bound B5 provides a poor estimate and is mostly out of scale. Parameters used: $a=a_{0}=3.0$, $b_{0}=1.0$, $g=0.7$, $\omega=1.2$, $N= 9$.
• Figure 2: The same as in Fig. 1 but for $\alpha=1/2$, corresponding to Hellinger distance. Again the bound B1 gives the best estimate, but the accuracy for some other bounds is different than in Fig. 1. Notably, the bounds B4 (diamonds) and B6 (pluses) provide often better estimates than the bound B2 (dotted line). The same labels used and parameters as in Fig. 1.