Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DNNs, Dataset Statistics, and Correlation Functions

Robert W. Batterman, James F. Woodward

TL;DR

This work argues that the remarkable generalization of DNNs in image tasks arises not solely from model capacity but from exploiting structured, higher-order correlations present in real-world datasets. It introduces a correlation-function methodology to connect mesoscale statistical structures in data to macroscopic, continuum-like behavior, drawing an analogy to representative volume elements in materials science. Empirical evidence from random matrix theory studies of datasets and weight matrices, plus demonstrations of higher-order correlations in MNIST, support the claim that DNNs learn and utilize N-point statistics beyond two-point covariances. The authors propose a research program focused on identifying and leveraging world-structure features that enable robust generalization, challenging purely worst-case SLT explanations. The practical impact is a shift toward data-centric explanations for generalization and toward methods that uncover and exploit dataset statistics at multiple scales.

Abstract

This paper argues that dataset structure is important in image recognition tasks (among other tasks). Specifically, we focus on the nature and genesis of correlational structure in the actual datasets upon which DNNs are trained. We argue that DNNs are implementing a widespread methodology in condensed matter physics and materials science that focuses on mesoscale correlation structures that live between fundamental atomic/molecular scales and continuum scales. Specifically, we argue that DNNs that are successful in image classification must be discovering high order correlation functions. It is well-known that DNNs successfully generalize in apparent contravention of standard statistical learning theory. We consider the implications of our discussion for this puzzle.

DNNs, Dataset Statistics, and Correlation Functions

TL;DR

This work argues that the remarkable generalization of DNNs in image tasks arises not solely from model capacity but from exploiting structured, higher-order correlations present in real-world datasets. It introduces a correlation-function methodology to connect mesoscale statistical structures in data to macroscopic, continuum-like behavior, drawing an analogy to representative volume elements in materials science. Empirical evidence from random matrix theory studies of datasets and weight matrices, plus demonstrations of higher-order correlations in MNIST, support the claim that DNNs learn and utilize N-point statistics beyond two-point covariances. The authors propose a research program focused on identifying and leveraging world-structure features that enable robust generalization, challenging purely worst-case SLT explanations. The practical impact is a shift toward data-centric explanations for generalization and toward methods that uncover and exploit dataset statistics at multiple scales.

Abstract

This paper argues that dataset structure is important in image recognition tasks (among other tasks). Specifically, we focus on the nature and genesis of correlational structure in the actual datasets upon which DNNs are trained. We argue that DNNs are implementing a widespread methodology in condensed matter physics and materials science that focuses on mesoscale correlation structures that live between fundamental atomic/molecular scales and continuum scales. Specifically, we argue that DNNs that are successful in image classification must be discovering high order correlation functions. It is well-known that DNNs successfully generalize in apparent contravention of standard statistical learning theory. We consider the implications of our discussion for this puzzle.

Paper Structure

This paper contains 11 sections, 26 equations, 12 figures.

Table of Contents

  1. Introduction
  2. Natural Images: Objects and Scaling
  3. A Correlation Function Methodology
  4. Dataset Statistics
  5. Scaling in Datasets
  6. RMT and the Statistics of Layer Weight Matrices During Training
  7. Recap
  8. Implementing the Correlation Function Methodology
  9. $N$-point Correlation Functions
  10. Learning Higher Order Correlations
  11. Conclusion

Figures (12)

  • Figure 1: Stream, Trees, Rocks
  • Figure 2: Blocking and averaging to yield a new (coarse-grained) effective system kad-handbook
  • Figure 3: Scaling of Contrast Distributionruderman-Bialek
  • Figure 4: Throwing Line Segments on the Plane. [10, p. 3393] (a) yields $p_0(x)$; (b) yields $p_1(x)$; (c) yields $p_2(x)$ .
  • Figure 5: Conductor/Insulator Composite. Dark bands are the Insulators.
  • ...and 7 more figures