Regularisation in neural networks: a survey and empirical analysis of approaches

TL;DR

Regularisation in neural networks addresses the challenge of generalisation, showing that common techniques do not universally improve performance. The authors provide a taxonomy dividing regularisation methods into data-based, architecture, training, and loss-function categories, and they empirically benchmark a subset of techniques across numeric and image tasks with MLP and CNN models. Key findings include strong dataset- and architecture-dependence of regularisation effects, with no one-size-fits-all solution; some methods like batch normalisation can help image tasks while others like weight perturbation often hurt. The study highlights notable contradictions (e.g., early stopping vs epoch-wise over-training) and synergies (e.g., data augmentation with noise-injection, dropout with pruning), underscoring the need to tailor regularisation choices to task characteristics and loss-landscape considerations such as double descent. These insights offer practical guidance for practitioners and lay groundwork for future systematic investigations into how dataset meta-features interact with regularisation to shape generalisation.

Abstract

Despite huge successes on a wide range of tasks, neural networks are known to sometimes struggle to generalise to unseen data. Many approaches have been proposed over the years to promote the generalisation ability of neural networks, collectively known as regularisation techniques. These are used as common practice under the assumption that any regularisation added to the pipeline would result in a performance improvement. In this study, we investigate whether this assumption holds in practice. First, we provide a broad review of regularisation techniques, including modern theories such as double descent. We propose a taxonomy of methods under four broad categories, namely: (1) data-based strategies, (2) architecture strategies, (3) training strategies, and (4) loss function strategies. Notably, we highlight the contradictions and correspondences between the approaches in these broad classes. Further, we perform an empirical comparison of the various regularisation techniques on classification tasks for ten numerical and image datasets applied to the multi-layer perceptron and convolutional neural network architectures. Results show that the efficacy of regularisation is dataset-dependent. For example, the use of a regularisation term only improved performance on numeric datasets, whereas batch normalisation improved performance on image datasets only. Generalisation is crucial to machine learning; thus, understanding the effects of applying regularisation techniques, and considering the connections between them is essential to the appropriate use of these methods in practice.

Figures (4)

• Figure 1: A tree representation of the proposed taxonomy of regularisation methods.
• Figure 2: Box and whisker charts of the normalized testing F1 scores for numeric datasets.
• Figure 3: Box and whisker charts of the testing F1 scores for various image datasets.
• Figure 4: Line charts of the testing F1 scores for various image datasets vs epochs.