Dataset-Free Weight-Initialization on Restricted Boltzmann Machine

TL;DR

The paper addresses dataset-free initialization for Bernoulli--Bernoulli RBMs by deriving a principled weight scale from a statistical-mechanical layer-correlation criterion. It uses the replica method under replica-symmetric assumptions to compute a free energy $f(\beta)$ and defines LC as the negative second bias derivative of that free energy, then selects $\beta_{\mathrm{max}}$ to maximize the LC. The proposed scheme draws weights from $\mathcal{N}(0, (\beta_{\mathrm{max}}/\sqrt{n+m})^2)$ and fixes biases to zero (visible) or a non-positive constant (hidden), recovering Xavier initialization in a special case. Empirical results on toy and real datasets (DB, ULC, MNIST) validate that initializing at $\beta_{\mathrm{max}}$ improves learning speed and final likelihood, with $\beta_{\mathrm{max}}$ depending only on $\alpha=m/n$, $c$, and $\mathcal{X}_h$. The work suggests a bridge between initialization theory and criticality, and outlines avenues for extending to GBRBMs and analytical beta_max expressions.

Abstract

In feed-forward neural networks, dataset-free weight-initialization methods such as LeCun, Xavier (or Glorot), and He initializations have been developed. These methods randomly determine the initial values of weight parameters based on specific distributions (e.g., Gaussian or uniform distributions) without using training datasets. To the best of the authors' knowledge, such a dataset-free weight-initialization method is yet to be developed for restricted Boltzmann machines (RBMs), which are probabilistic neural networks consisting of two layers. In this study, we derive a dataset-free weight-initialization method for Bernoulli--Bernoulli RBMs based on statistical mechanical analysis. In the proposed weight-initialization method, the weight parameters are drawn from a Gaussian distribution with zero mean. The standard deviation of the Gaussian distribution is optimized based on our hypothesis that a standard deviation providing a larger layer correlation (LC) between the two layers improves the learning efficiency. The expression of the LC is derived based on a statistical mechanical analysis. The optimal value of the standard deviation corresponds to the maximum point of the LC. The proposed weight-initialization method is identical to Xavier initialization in a specific case (i.e., when the sizes of the two layers are the same, the random variables of the layers are $\{-1,1\}$-binary, and all bias parameters are zero). The validity of the proposed weight-initialization method is demonstrated in numerical experiments using a toy and real-world datasets.

Figures (4)

• Figure 1: Dependency of $|\chi_{ \mathrm{v} , \mathrm{h} }|$ on $\beta$ when $\mathcal{X} _{ \mathrm{h} } = \mathcal{B}$: (a) $\alpha = 0.5$, (b) $\alpha = 1$, and (c) $\alpha = 2$.
• Figure 2: Dependency of $|\chi_{ \mathrm{v} , \mathrm{h} }|$ on $\beta$ when $\mathcal{X} _{ \mathrm{h} } = \mathcal{I}$.
• Figure 3: Four base patterns with $n = 20$ elements: (a) all are $1$, (b) all are $-1$, (c) the fist 10 elements are $1$ and the others are $-1$, and (d) the reverse of (c).
• Figure 4: Log-likelihood differences on a long-term learning. The square and circle points denote the log likelihood with $\beta = \beta_{ \mathrm{max} }$ minus that with $\beta = 4\beta_{ \mathrm{max} }$ and with $\beta = \beta_{ \mathrm{max} }/4$, respectively. These points denote the average over 100 experiments.