AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks
Bruno Belucci, Karim Lounici, Katia Meziani
TL;DR
The paper tackles the fragility of neural networks in small-tabular regression by introducing AdaCap, a training scheme that couples a Tikhonov-regularized closed-form output mapping with a permutation-based contrastive loss. The method defines $W(lambda) = (H^T H + lambda I_d)^{-1} H^T Y$ and uses the same mapping to compute predictions, while contrasting fits on true labels with fits on permuted labels to enforce genuine input–target dependencies; lambda is initialized via a forward pass over a log-grid. Empirically, AdaCap yields consistent improvements across 85 real-world datasets, particularly for residual architectures in the small-data regime, and a meta-classifier can predict when AdaCap will help with about 70% accuracy based on dataset size, skewness, and noise. This work demonstrates that AdaCap acts as a targeted, interpretable regularizer whose benefits are predictable, and it provides public code to enable broad adoption.
Abstract
Neural networks struggle on small tabular datasets, where tree-based models remain dominant. We introduce Adaptive Contrastive Approach (AdaCap), a training scheme that combines a permutation-based contrastive loss with a Tikhonov-based closed-form output mapping. Across 85 real-world regression datasets and multiple architectures, AdaCap yields consistent and statistically significant improvements in the small-sample regime, particularly for residual models. A meta-predictor trained on dataset characteristics (size, skewness, noise) accurately anticipates when AdaCap is beneficial. These results show that AdaCap acts as a targeted regularization mechanism, strengthening neural networks precisely where they are most fragile. All results and code are publicly available at https://github.com/BrunoBelucci/adacap.