Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting

TL;DR

This paper tackles overfitting in deep learning models used for software engineering tasks by introducing OverfitGuard, a history-based method that leverages time-series classifiers trained on training histories to both detect overfitting in trained models and to guide early stopping during training. It trains on simulated histories and validates on real-world samples, achieving an F1 score of 0.91 for detection and enabling stopping earlier than traditional early stopping in many cases without sacrificing the best-model recovery. The approach is non-intrusive, avoiding modifications to model architecture or data while using the natural learning curves generated during training. The authors provide a replication package and demonstrate practical benefits for improving DL software quality, while also acknowledging computational cost and calling for broader real-world data and online-update extensions.

Abstract

In software engineering, deep learning models are increasingly deployed for critical tasks such as bug detection and code review. However, overfitting remains a challenge that affects the quality, reliability, and trustworthiness of software systems that utilize deep learning models. Overfitting can be (1) prevented (e.g., using dropout or early stopping) or (2) detected in a trained model (e.g., using correlation-based approaches). Both overfitting detection and prevention approaches that are currently used have constraints (e.g., requiring modification of the model structure, and high computing resources). In this paper, we propose a simple, yet powerful approach that can both detect and prevent overfitting based on the training history (i.e., validation losses). Our approach first trains a time series classifier on training histories of overfit models. This classifier is then used to detect if a trained model is overfit. In addition, our trained classifier can be used to prevent overfitting by identifying the optimal point to stop a model's training. We evaluate our approach on its ability to identify and prevent overfitting in real-world samples. We compare our approach against correlation-based detection approaches and the most commonly used prevention approach (i.e., early stopping). Our approach achieves an F1 score of 0.91 which is at least 5% higher than the current best-performing non-intrusive overfitting detection approach. Furthermore, our approach can stop training to avoid overfitting at least 32% of the times earlier than early stopping and has the same or a better rate of returning the best model.

Figures (6)

• Figure 1: Examples of overfit and non-overfit training histories.
• Figure 2: Early stopping and our approach for overfitting prevention.
• Figure 3: Our approach for overfitting detection and prevention.
• Figure 4: Overview of the experimental setup.
• Figure 5: The optimal rate of our overfitting prevention approach (using a rolling window) and early stopping with different patience values.