Implicit Hypothesis Testing and Divergence Preservation in Neural Network Representations

TL;DR

The paper reframes supervised classification as a sequence of binary hypothesis tests on learned representations, linking neural network decision rules to Neyman–Pearson optimality and to KL-divergence–driven error exponents. It develops a theoretical bridge via Stein’s lemma and the data-processing inequality, introducing the evidence–error plane to separate representation quality (divergence retained) from decision efficiency (error exponent). Empirically, it demonstrates NP-like behavior on synthetic Gaussian data and real-world datasets (Binary Image, Yin–Yang, MNIST) and analyzes dynamics in fully connected nets and Spiking Neural Networks, including the impact of majority voting. The framework suggests practical regularization and ensemble strategies anchored in information-theoretic objectives and highlights avenues for extending the analysis to Chernoff information and Bayesian error regimes, with implications for safer, quantifiable deployment of neural classifiers.

Abstract

We study the supervised training dynamics of neural classifiers through the lens of binary hypothesis testing. We model classification as a set of binary tests between class-conditional distributions of representations and empirically show that, along training trajectories, well-generalizing networks increasingly align with Neyman-Pearson optimal decision rules via monotonic improvements in KL divergence that relate to error rate exponents. We finally discuss how this yields an explanation and possible training or regularization strategies for different classes of neural networks.

Figures (8)

• Figure 1: 4D Gaussian Dataset Error Rates: Green region is all achievable error rates and the black cruve defines the NP-optimal envelope. Each point is the empirical error rates of the network on the corresponding epoch. The network converges to the Bayes optimal classifier's error rate, which is the intersection between the curve and the dotted line $\beta = \alpha$. See \ref{['Gaussian Experiment']}.
• Figure 2: Sample points from the Binary Image Dataset
• Figure 3: DNN trained on Binary Image: The network improves the KL divergence between its class conditioned representations as epochs increase and effectively exploits that divergence to reach the Stein error regime. This is the prototypical NP optimal neural network. The solid blue line denotes $D_{inp}$ for this dataset and the dotted blue line is the predicted error rates from Stein's Lemma.
• Figure 4: DNN trained on MNIST: The network trained on MNIST. A similar trajectory to Figure \ref{['fig:evidence-error-binary-image']} is observed, although the network operates noticeably above the Stein limit.
• Figure 5: Majority Voting on Information Inefficient Network: A smaller network with only two hidden layers, trained on the Yin-Yang dataset and evaluated with majority voting classification. As the number of samples $n$ used in the vote increase, the network performance also increases at the same classifier KL divergence level.