Statistical Mechanics and Artificial Neural Networks: Principles, Models, and Applications

TL;DR

The first part of this chapter provides an overview of the principles, models, and applications of ANNs, highlighting their connections to statistical mechanics and statistical learning theory.

Abstract

The field of neuroscience and the development of artificial neural networks (ANNs) have mutually influenced each other, drawing from and contributing to many concepts initially developed in statistical mechanics. Notably, Hopfield networks and Boltzmann machines are versions of the Ising model, a model extensively studied in statistical mechanics for over a century. In the first part of this chapter, we provide an overview of the principles, models, and applications of ANNs, highlighting their connections to statistical mechanics and statistical learning theory. Artificial neural networks can be seen as high-dimensional mathematical functions, and understanding the geometric properties of their loss landscapes (i.e., the high-dimensional space on which one wishes to find extrema or saddles) can provide valuable insights into their optimization behavior, generalization abilities, and overall performance. Visualizing these functions can help us design better optimization methods and improve their generalization abilities. Thus, the second part of this chapter focuses on quantifying geometric properties and visualizing loss functions associated with deep ANNs.

Figures (12)

• Figure 1: A schematic of a perceptron with output $y$, inputs $x_1, x_2, \dots, x_n$, corresponding weights $w_1, w_2, \dots, w_n$, and a bias term $b$. Traditionally, the inputs and outputs of perceptrons were considered to be binary (e.g., $\{-1, 1\}$ or $\{0, 1\}$), while weights and biases are generally represented as real numbers. The symbol $\Sigma$ signifies the summation process wherein the artificial neuron computes $\sum_{i}w_i x_i + b$. This computed value then becomes the input for a step activation function, denoted by a step symbol. We denote the final output as $y$. While the original perceptron employed a step activation function, contemporary applications frequently utilize a variety of functions including sigmoid, $\tanh$, and rectified linear unit (ReLU).
• Figure 2: An example of a Hopfield network with $n=5$ neurons. Each neuron is connected to all other neurons through black edges, representing both inputs and outputs. (Adapted from martin_thoma_github.)
• Figure 3: The sigmoid function $\sigma_i\equiv \sigma (\Delta E_i/T)$ [see Eq. \ref{['eq:boltzmann_machine']}] as function of $\Delta E_i$ for $T=0.5,1,2$.
• Figure 4: Boltzmann machines consist of visible units (blue) and hidden units (red). In the shown example, there are five visible units $\left\{v_i\right\}$$({i\in\{1,\dots,5\}})$ and three hidden units $\left\{h_j\right\}$$(j\in\{1,2,3\})$. Similar to Hopfield networks, the network architecture in a Boltzmann machine is complete.
• Figure 5: Restricted Boltzmann machines consist of a visible layer (blue) and a hidden layer (red). In the shown example, the respective layers comprise six visible units $\left\{v_i\right\}$$(i\in\{1,\dots,6\})$ and four hidden units $\left\{h_j\right\}$$(j\in\{1,\dots,4\})$. The network structure of an RBM is bipartite and undirected.