Random Linear Projections Loss for Hyperplane-Based Optimization in Neural Networks

TL;DR

This work introduces Random Linear Projections (RLP) loss, a hyperplane-based, non-local objective that minimizes the distance between regression hyperplanes derived from fixed-size subsets of features and labels. The authors prove that the RLP optimizer targets the conditional expectation $h(x)=\mathbb{E}[Y|X=x]$ and show faster convergence than MSE under suitable assumptions, presenting a two-step algorithm: balanced batch generation and RLP-based training. Empirically, RLP improves performance across regression, image reconstruction, and classification tasks (e.g., California Housing, MNIST, CIFAR-10), demonstrating faster convergence, better generalization, and robustness to limited data, distribution shifts, and additive noise; they also explore mixup variants. A key caveat is the computational cost from matrix inversions at each step, highlighting directions for scalable optimization and further theoretical development to solidify the statistical properties of RLP losses.

Abstract

Advancing loss function design is pivotal for optimizing neural network training and performance. This work introduces Random Linear Projections (RLP) loss, a novel approach that enhances training efficiency by leveraging geometric relationships within the data. Distinct from traditional loss functions that target minimizing pointwise errors, RLP loss operates by minimizing the distance between sets of hyperplanes connecting fixed-size subsets of feature-prediction pairs and feature-label pairs. Our empirical evaluations, conducted across benchmark datasets and synthetic examples, demonstrate that neural networks trained with RLP loss outperform those trained with traditional loss functions, achieving improved performance with fewer data samples, and exhibiting greater robustness to additive noise. We provide theoretical analysis supporting our empirical findings.

Figures (23)

• Figure 1: Comparing true and predicted functions: illustration that two functions are equivalent iff they share identical hyperplanes generated by all possible feature-label pairs.
• Figure 2: Test performance comparison across six datasets (California Housing, Wine Quality, Linear, Nonlinear, MNIST, and CIFAR-10) using three different loss functions: Mean Squared Error (MSE), MSE with $L_2$ regularization (MSE + $L_2$), and RLP. The x-axis represents training epochs, while the y-axis indicates the test MSE.
• Figure 3: Distribution shift test performance comparison across three datasets (California Housing, Wine Quality, and Nonlinear) using three different loss functions: Mean Squared Error (MSE), MSE with $L_2$ regularization (MSE + $L_2$), and RLP. The x-axis is the degree of bias, $\gamma$, between the test data and the train data, while the y-axis indicates the test MSE.
• Figure 4: Test performance comparison on MNIST using Cross Entropy loss and RLP loss. The x-axis represents training epochs, while the y-axis indicates the classification accuracy (left) and F1 score (right).
• Figure 5: Comparison of reconstructed images for an autoencoder trained with MSE loss (top row) and RLP loss (bottom row) at different epochs. The model trained with RLP loss learns faster and better with limited data ($|J| = 50$).

Theorems & Definitions (6)

• Definition 2.1: MSE Loss
• Definition 2.2: Random Linear Projections Loss
• Proposition 2.3
• Proposition 2.4
• proof
• proof