Conjugate Learning Theory: Uncovering the Mechanisms of Trainability and Generalization in Deep Neural Networks

TL;DR

Conjugate Learning Theory offers a unified, duality-based framework to analyze trainability and generalization in deep neural networks under finite data. By casting learning as conditional distribution estimation within exponential-family constraints and employing Fenchel–Young losses, the approach links empirical-risk optimization to gradient energy and the spectrum of a model-induced structure matrix, while bounding generalization via generalized conditional entropy and information-loss measures. The theory yields data-determined lower bounds on empirical risk and reveals how architecture (depth, width, skip connections) and optimization choices shape trainability and generalization in a principled way. Empirical validation across LeNet, ResNet-18, and ViT on MNIST and CIFAR demonstrates that the predicted relationships between gradient energy, eigenvalues, and loss bounds hold in practice, supporting the framework's relevance for real-world DNN training.

Abstract

In this work, we propose a notion of practical learnability grounded in finite sample settings, and develop a conjugate learning theoretical framework based on convex conjugate duality to characterize this learnability property. Building on this foundation, we demonstrate that training deep neural networks (DNNs) with mini-batch stochastic gradient descent (SGD) achieves global optima of empirical risk by jointly controlling the extreme eigenvalues of a structure matrix and the gradient energy, and we establish a corresponding convergence theorem. We further elucidate the impact of batch size and model architecture (including depth, parameter count, sparsity, skip connections, and other characteristics) on non-convex optimization. Additionally, we derive a model-agnostic lower bound for the achievable empirical risk, theoretically demonstrating that data determines the fundamental limit of trainability. On the generalization front, we derive deterministic and probabilistic bounds on generalization error based on generalized conditional entropy measures. The former explicitly delineates the range of generalization error, while the latter characterizes the distribution of generalization error relative to the deterministic bounds under independent and identically distributed (i.i.d.) sampling conditions. Furthermore, these bounds explicitly quantify the influence of three key factors: (i) information loss induced by irreversibility in the model, (ii) the maximum attainable loss value, and (iii) the generalized conditional entropy of features with respect to labels. Moreover, they offer a unified theoretical lens for understanding the roles of regularization, irreversible transformations, and network depth in shaping the generalization behavior of deep neural networks. Extensive experiments validate all theoretical predictions, confirming the framework's correctness and consistency.

Figures (11)

• Figure 1: Schematic illustration of the conjugate learning framework. The diagram outlines the complete processing pipeline from raw input to learning target approximation, emphasizing the interplay among model output, conjugate transformation, and distance measurement.
• Figure 2: Schematic illustration of the mapping between gradient energy and the global optimum of the empirical risk. The top subplot shows the gradient energy as a function of the model parameters $\theta$, with blue dots indicating points where the gradient energy is zero. The bottom subplot depicts the corresponding empirical risk landscape, with its global minimum marked by a pentagram. For positive definite structure matrices, zero gradient energy points coincide with the global minimum of the empirical risk, validating gradient energy minimization as a proxy for empirical risk reduction.
• Figure 3: Custom-designed model architectures and configuration parameters. Gray blocks represent components where the number of repetitions can be adjusted via the parameter $n_d$, and model width can be tuned via the parameter $n_w$. Model B is a modified variant of Model A with additional skip connections. The symbol $I$ denotes an identity transformation, which preserves the dimensionality of feature maps in skip connection pathways.
• Figure 4: Training dynamics of LeNet with softmax cross entropy loss.
• Figure 5: Training dynamics of LeNet with MSE loss.

Theorems & Definitions (45)

• Theorem 1: Sufficiency Principle
• Theorem 2: Pitman--Darmois--Koopmans
• Lemma 3: Properties of Convex Conjugate Duality
• Lemma 4: Properties of Fenchel--Young Losses Blondel2019LearningWF
• Lemma 5: KL Divergence Upper Bound
• Lemma 6: Pinsker's Inequality 2017Elements
• Proposition 7: Practical Learnability
• definition 1: Conjugate Learning
• Theorem 8
• Theorem 9