Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Model

Onat Ure, Samet Demir, Zafer Dogan

TL;DR

This paper tackles how high-order statistics beyond Gaussianity influence neural network learning. By introducing a cumulant-controllable data model using a two-layer generator with Hermite-expanded activations, the authors can independently modulate cumulants such as skewness and kurtosis via finite Hermite coefficients $\{c_i\}$. They demonstrate, through synthetic and Fashion-MNIST experiments, that networks exhibit moment-wise learning: they first fit low-order moments (mean and covariance) and progressively leverage higher-order cumulants to improve generalization. The approach bridges simplistic Gaussian assumptions and complex real-world data, offering a principled tool to study distributional effects in learning and signal processing with clear controllability and expressivity.

Abstract

We study the effect of high-order statistics of data on the learning dynamics of neural networks (NNs) by using a moment-controllable non-Gaussian data model. Considering the expressivity of two-layer neural networks, we first construct the data model as a generative two-layer NN where the activation function is expanded by using Hermite polynomials. This allows us to achieve interpretable control over high-order cumulants such as skewness and kurtosis through the Hermite coefficients while keeping the data model realistic. Using samples generated from the data model, we perform controlled online learning experiments with a two-layer NN. Our results reveal a moment-wise progression in training: networks first capture low-order statistics such as mean and covariance, and progressively learn high-order cumulants. Finally, we pretrain the generative model on the Fashion-MNIST dataset and leverage the generated samples for further experiments. The results of these additional experiments confirm our conclusions and show the utility of the data model in a real-world scenario. Overall, our proposed approach bridges simplified data assumptions and practical data complexity, which offers a principled framework for investigating distributional effects in machine learning and signal processing.

Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Model

TL;DR

This paper tackles how high-order statistics beyond Gaussianity influence neural network learning. By introducing a cumulant-controllable data model using a two-layer generator with Hermite-expanded activations, the authors can independently modulate cumulants such as skewness and kurtosis via finite Hermite coefficients . They demonstrate, through synthetic and Fashion-MNIST experiments, that networks exhibit moment-wise learning: they first fit low-order moments (mean and covariance) and progressively leverage higher-order cumulants to improve generalization. The approach bridges simplistic Gaussian assumptions and complex real-world data, offering a principled tool to study distributional effects in learning and signal processing with clear controllability and expressivity.

Abstract

We study the effect of high-order statistics of data on the learning dynamics of neural networks (NNs) by using a moment-controllable non-Gaussian data model. Considering the expressivity of two-layer neural networks, we first construct the data model as a generative two-layer NN where the activation function is expanded by using Hermite polynomials. This allows us to achieve interpretable control over high-order cumulants such as skewness and kurtosis through the Hermite coefficients while keeping the data model realistic. Using samples generated from the data model, we perform controlled online learning experiments with a two-layer NN. Our results reveal a moment-wise progression in training: networks first capture low-order statistics such as mean and covariance, and progressively learn high-order cumulants. Finally, we pretrain the generative model on the Fashion-MNIST dataset and leverage the generated samples for further experiments. The results of these additional experiments confirm our conclusions and show the utility of the data model in a real-world scenario. Overall, our proposed approach bridges simplified data assumptions and practical data complexity, which offers a principled framework for investigating distributional effects in machine learning and signal processing.
Paper Structure (7 sections, 2 theorems, 4 equations, 2 figures)

This paper contains 7 sections, 2 theorems, 4 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Proposed Framework
  3. EXPERIMENTAL RESULTS
  4. Experimental setting
  5. Experiments with synthetic data
  6. Experiments on Fashion-MNIST
  7. CONCLUSION

Key Result

Proposition 1

Suppose hermite_exp_theta holds and $p>d$. Then, the class of data models defined by data_model is dense in the space of $d$-dimensional probability measures with finite moments under standard weak topologies (e.g., Wasserstein metrics).

Figures (2)

  • Figure 1: Gaussian vs. non-Gaussian binary classification (synthetic data): Test losses on the non-Gaussian dataset and its Gaussian-equivalent counterpart with matched mean and covariance (Sec \ref{['sec:exp-setting']}) for a two-layer neural network trained to discriminate between standard Gaussian and non-Gaussian data \ref{['final_data_model']}. For the non-Gaussian data model \ref{['final_data_model']}, we set parameters $\mathbf{W} = \mathbf{F} = \mathbf{I}$, $\boldsymbol{\mu} = \mathbf{0}$, $\boldsymbol{\Sigma} = {\mathbf I}_p$, while the degree of Hermite expansion is $\ell = 3$, the latent dimension is $p = 128$ and the Hermite coefficients are chosen as $c_0 = 0.4$, and $c_1 = 0.5$. The considered two-layer neural network has $512$ hidden units and ReLU as activation function. The network is optimized using online SGD with mean-squared error loss, and the learning rate of $10^{-1}$. Results are averaged over 5 independent experiments.
  • Figure 2: Gaussian vs. non-Gaussian binary classification (Fashion-MNIST data): Test losses on the non-Gaussian dataset and its Gaussian-equivalent counterpart with matched mean and covariance (Sec \ref{['sec:exp-setting']} and \ref{['sec:mnist']}) for a two-layer neural network trained to discriminate between standard Gaussian and non-Gaussian Fashion-MNIST samples generated by our pretrained data model \ref{['final_data_model']}. For this experiment, we focus only on the samples of the T-shirt/top class (label 0). To train the parameters of the data model, we consider a GAN model with a generator of the form ${\mathbf W} \tanh(\mathbf{Fz} + \mathbf{b})$, where only $\mathbf{F}$ and $\mathbf{b}$ are trained to produce realistic T-shirt samples while ${\mathbf W}$ is set to identity and a discriminator is a two-layer ReLU network with a final sigmoid output. Both generator and discriminator are trained adversarially for 100 epochs using Adam (learning rate $10^{-4}$, $\beta_1=0.9$, $\beta_2=0.999$), batch size 48, and binary cross-entropy loss. After the training of the GAN goodfellow2014generative, we apply Hermite expansion (truncated at degree $\ell =5$) to $\tanh$ and use the resulting Hermite coefficients together with the parameters ${\mathbf W}, {\mathbf F}, \mathbf{b}$ to reach our cumulant-controllable data model \ref{['final_data_model']}. For the binary classification, the two-layer neural network is considered with $512$ hidden units, and the latent dimension is $p =100$. The model is optimized via online stochastic gradient descent (SGD) with a mean squared error loss and a fixed learning rate of $3 \times 10^{-4}$.

Theorems & Definitions (4)

  • Proposition 1: Expressivity
  • proof : Proof (Concept)
  • Proposition 2: Controllability of cumulants for finite $\ell$
  • proof