Evaluating the Impact of Data Augmentation on Predictive Model Performance

TL;DR

The paper addresses data scarcity and reproducibility in learning analytics by replicating a prior enrollment-prediction study and rigorously evaluating data augmentation techniques. It re-trains four standard ML models on a replicated dataset, examining 21 augmentation methods (and 99 chained configurations) with bootstrapped AUC and multiple testing correction. The main finding is that SMOTE-ENN sampling, especially when combined with added noise, provides the most reliable predictive gains and often reduces training time, while deep generative methods offer limited or unstable improvements. These results offer practical guidance for LA researchers on when and how augmentation helps, and the work strengthens the field's credibility by providing open-source code and a validated replication framework.

Abstract

In supervised machine learning (SML) research, large training datasets are essential for valid results. However, obtaining primary data in learning analytics (LA) is challenging. Data augmentation can address this by expanding and diversifying data, though its use in LA remains underexplored. This paper systematically compares data augmentation techniques and their impact on prediction performance in a typical LA task: prediction of academic outcomes. Augmentation is demonstrated on four SML models, which we successfully replicated from a previous LAK study based on AUC values. Among 21 augmentation techniques, SMOTE-ENN sampling performed the best, improving the average AUC by 0.01 and approximately halving the training time compared to the baseline models. In addition, we compared 99 combinations of chaining 21 techniques, and found minor, although statistically significant, improvements across models when adding noise to SMOTE-ENN (+0.014). Notably, some augmentation techniques significantly lowered predictive performance or increased performance fluctuation related to random chance. This paper's contribution is twofold. Primarily, our empirical findings show that sampling techniques provide the most statistically reliable performance improvements for LA applications of SML, and are computationally more efficient than deep generation methods with complex hyperparameter settings. Second, the LA community may benefit from validating a recent study through independent replication.

Figures (1)

• Figure 1: Conceptual illustration of the research methods and steps, along with the relationship between the two RQs.