Evolution of Conditional Entropy for Diffusion Dynamics on Graphs

TL;DR

This work introduces the conditional entropy of heat diffusion in graphs and demonstrates that this entropic measure satisfies the first and second laws of thermodynamics, thereby providing a physical interpretation of diffusion dynamics on networks.

Abstract

The modeling of diffusion processes on graphs is the basis for many network science and machine learning approaches. Entropic measures of network-based diffusion have recently been employed to investigate the reversibility of these processes and the diversity of the modeled systems. While results about their steady state are well-known, very few exact results about their time evolution exist. Here, we introduce the conditional entropy of heat diffusion in graphs. We demonstrate that this entropic measure satisfies the first and second laws of thermodynamics, thereby providing a physical interpretation of diffusion dynamics on networks. We outline a mathematical framework that contextualizes diffusion and conditional entropy within the theories of continuous-time Markov chains and information theory. Furthermore, we obtain explicit results for its evolution on complete, path, and circulant graphs, as well as a mean-field approximation for Erdös-Rényi graphs. We also obtain asymptotic results for general networks. Finally, we experimentally demonstrate several properties of conditional entropy for diffusion over random graphs, such as the Watts-Strogatz model.

Figures (4)

• Figure 1: Heat diffusion on a path graph with 10 nodes with initial condition $\boldsymbol{\mathbf{p}}(0) = \delta_1$. (A) Initially, all the transition probability is concentrated on the source node itself; over time, the heat diffuses. At stationarity, transition to any node is equally likely, but convergence to the limit value is not necessarily monotonic. (B) Conditional entropy of the process conditioned on the $\delta_1$ initial condition. The line shows the evolution of the conditional entropy on a linear time scale. The entropy starts at a value 0 and grows monotonically towards an asymptotic value, which is reached at stationarity. (C) The line shows the same entropy curve but on a logarithmic time scale and displays a typical logistic-like growth.
• Figure 2: Conditional entropy for heat diffusion on a complete graph, path graph, and circulant graphs with different step-sets. The entropy curves are bounded from below by the path graph curve and above by the complete graph curve. (A) Three circulant graphs with increasing density. The density of $C_{20}(\{1\})$ is equal to $\frac{2}{19}$, the one of $C_{20}(\{1, 2\})$ is $\frac{4}{19}$, and the one of $C_{20}(\{1, 2, 3\})$ is $\frac{6}{19}$. The entropy grows faster in denser graphs. (B) Three circulant graphs with the same density but decreasing diameter. The diameter of $C_{20}(\{1\})$ is equal to $10$, the one of $C_{20}(\{1, 2\})$ is $5$, and the one of $C_{20}(\{1, 2, 3\})$ is $4$.The entropy grows faster as the diameter decreases.
• Figure 3: Comparison of Watts-Strogatz and ER graphs with circulant matrices sharing similar structural properties. For all stochastic models, we sample $10$ graphs of size $100$. (A) We use as reference $C_{100}(S)$, with $S = \{ 1,2,3 \}$. We use it as a skeleton for obtaining three Watts-Strogatz network models with increasing probability of edge rewiring: $p= 0.1, 0.25, 0.5$. The plot shows that by rewiring more edges, we raise the speed at which the conditional entropy increases. For high numbers of rewired edges, the variance of the entropy curve is low (shaded area indicates $\pm 1$ standard deviation). (B) We compare the conditional entropy averaged on a set of ER graphs to that of circulant graphs $G_1 = C_{100}(\{ 1,2,3,4, 13 \}), G_2 = C_{100}(\{ 1,3,5,11,13 \}), G_3 = C_{100}(\{ 1,10,11,12, 13 \})$ having a similar density to that of the ER graphs, but having other different structural properties. Indeed, the initial shape of the curve is similar because the number of neighbors is similar (by design). On the other hand, the conditional entropy on ER graphs then increases faster than on the circulant graphs because of the higher number of nodes at distance two compared to comparable circulant graphs. $G_1$ and $G_2$ have both diameters equal to $5$, but in the latter, nodes have fewer nodes at distances $4$ and $5$, whereas more at distances $2$ and $3$, leading to a faster growth of conditional entropy. In $G_3$, nodes have fewer nodes at distances $2$ compared to $G_2$, but more at distances $3$ and $4$, which is its diameter. Therefore, its entropy grows more slowly initially, but then catches up and overtakes that of $G_2$.
• Figure 4: Evolution of the conditional entropy over samples of 5 ER graphs (dotted lines) with three different density values plotted together with their mean-field approximation (dashed-dotted lines). (A) shows graphs of size 100, (B) graphs of size 500. The mean-field approximation overestimates entropy values; however, one can see that it becomes more accurate as the density of the graph and the size of the ER graph increase.

Theorems & Definitions (47)

• Definition 2.1
• Definition 2.2
• Definition 2.3
• Definition 2.4
• Proposition 2.5: Chain Rule for KL Divergence
• Proposition 2.6: Pinsker Inequality
• Definition 3.1
• proof
• Proposition 3.3
• proof