Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to be Reproducible: Custom Loss Design for Robust Neural Networks

Waqas Ahmed, Sheeba Samuel, Kevin Coakley, Birgitta Koenig-Ries, Odd Erik Gundersen

TL;DR

The paper tackles the problem of unreliable DL performance across runs due to randomness in initialization, shuffling, and nondeterministic operations. It proposes a Custom Loss Function (CLF) that combines Cross-Entropy with a Stable Loss and a Variance Penalty Loss to stabilize training and reduce run-to-run variability, defined as $L(\\theta; x, y) = CEL(\\theta; x, y) + \\lambda_s SL(\\theta; x, y) + \\lambda_v VPL(\\theta; x, y)$. Across CIFAR-10/100 and ETTh1 datasets with diverse architectures, CLF consistently reduces the standard deviation of performance metrics and MAE, often with larger gains for high-variance models and longer training exposure. These results suggest CLF as a lightweight, generalizable strategy to improve trustworthiness and robustness of neural networks in practical deployment.

Abstract

To enhance the reproducibility and reliability of deep learning models, we address a critical gap in current training methodologies: the lack of mechanisms that ensure consistent and robust performance across runs. Our empirical analysis reveals that even under controlled initialization and training conditions, the accuracy of the model can exhibit significant variability. To address this issue, we propose a Custom Loss Function (CLF) that reduces the sensitivity of training outcomes to stochastic factors such as weight initialization and data shuffling. By fine-tuning its parameters, CLF explicitly balances predictive accuracy with training stability, leading to more consistent and reliable model performance. Extensive experiments across diverse architectures for both image classification and time series forecasting demonstrate that our approach significantly improves training robustness without sacrificing predictive performance. These results establish CLF as an effective and efficient strategy for developing more stable, reliable and trustworthy neural networks.

Learning to be Reproducible: Custom Loss Design for Robust Neural Networks

TL;DR

The paper tackles the problem of unreliable DL performance across runs due to randomness in initialization, shuffling, and nondeterministic operations. It proposes a Custom Loss Function (CLF) that combines Cross-Entropy with a Stable Loss and a Variance Penalty Loss to stabilize training and reduce run-to-run variability, defined as . Across CIFAR-10/100 and ETTh1 datasets with diverse architectures, CLF consistently reduces the standard deviation of performance metrics and MAE, often with larger gains for high-variance models and longer training exposure. These results suggest CLF as a lightweight, generalizable strategy to improve trustworthiness and robustness of neural networks in practical deployment.

Abstract

To enhance the reproducibility and reliability of deep learning models, we address a critical gap in current training methodologies: the lack of mechanisms that ensure consistent and robust performance across runs. Our empirical analysis reveals that even under controlled initialization and training conditions, the accuracy of the model can exhibit significant variability. To address this issue, we propose a Custom Loss Function (CLF) that reduces the sensitivity of training outcomes to stochastic factors such as weight initialization and data shuffling. By fine-tuning its parameters, CLF explicitly balances predictive accuracy with training stability, leading to more consistent and reliable model performance. Extensive experiments across diverse architectures for both image classification and time series forecasting demonstrate that our approach significantly improves training robustness without sacrificing predictive performance. These results establish CLF as an effective and efficient strategy for developing more stable, reliable and trustworthy neural networks.
Paper Structure (18 sections, 14 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 18 sections, 14 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Custom loss function
  4. Cross-Entropy Loss (CEL)
  5. Stable Loss (SL)
  6. Variance penalty loss (VPL)
  7. Gradient of VPL.
  8. Methodology
  9. Hyperparameter optimization with Optuna:
  10. Experimental setup
  11. Results and discussion
  12. Impact of CLF on image classification
  13. Sample-based evaluation of CLF robustness
  14. CLF under overfitting conditions
  15. Impact of CLF activation duration on training performance
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Methodological overview (classification task): This diagram illustrates the process of fixed identical training conducted twenty times (n=20) to evaluate the efficacy of CLF on different set of dataset and models. We performed deterministic training by controlling for random sources. Model variability is assessed in terms of the standard deviation of accuracy and cross-entropy loss, aiming to quantify the robustness of the training process.
  • Figure 2: Test accuracy distribution across 5 runs for different CLF activation durations on CIFAR-10 with ResNet-14. The x-axis indicates the total number of epochs during which the CLF was active out of a fixed 500-epoch training schedule.