Information flow in multilayer perceptrons: an in-depth analysis

TL;DR

In this position article a specific investigation is conducted on the way information is processed, with particular reference to the requirements imposed by supervised learning, and the concept of information matrix is devised and used as formal framework for understanding the aetiology of optimisation strategies.

Abstract

Analysing how information flows along the layers of a multilayer perceptron is a topic of paramount importance in the field of artificial neural networks. After framing the problem from the point of view of information theory, in this position article a specific investigation is conducted on the way information is processed, with particular reference to the requirements imposed by supervised learning. To this end, the concept of information matrix is devised and then used as formal framework for understanding the aetiology of optimisation strategies and for studying the information flow. The underlying research for this article has also produced several key outcomes: i) the definition of a parametric optimisation strategy, ii) the finding that the optimisation strategy proposed in the information bottleneck framework shares strong similarities with the one derived from the information matrix, and iii) the insight that a multilayer perceptron serves as a kind of "adaptor", meant to process the input according to the given objective.

Figures (8)

• Figure 1: Notational conventions for denoting entropies and mutual information useful for analysing the relation between an input source $X$ and a target $Y$. $\mathcal{N}_{XY}$(standing for noise) and $L_{XY}$(standing for lack/loss of information) are both conditional entropies.
• Figure 2: Notational conventions for denoting significant entropies and mutual information useful for analysing the relation between $X\xspace \rightarrow Y\xspace$ throughout a transformation $F_{}$ that stands in between.
• Figure 3: Underlying dynamics of $\Psi$, in which both noise and loss flow from right to left, the former at the irrelevant side and the latter at the relevant side. Note that, in principle, the amount of noise and loss may also depend on a non-deterministic behaviour of $F_{}$.
• Figure 4: Noise-loss diagram reporting in a Cartesian space the main behavioural patterns found while analysing $\Psi$.
• Figure 5: Isometrics of $\mathcal{I}_{XX}$ reported in a noise-loss diagram. The figure highlights that all isometrics are in fact diagonal lines at 45 degrees.