Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-tuning with Very Large Dropout

Jianyu Zhang, Léon Bottou

TL;DR

This paper tackles the problem of out-of-distribution generalization when fine-tuning pre-trained models. It introduces a simple yet effective approach: applying very large dropout (around 90%) to the penultimate layer during fine-tuning to force the model to rely on a richer, more redundant set of features. Empirical results across multiple DomainBed tasks show that very-large dropout outperforms ensemble and weight-averaging baselines, and remains compatible with other fine-tuning tricks, while preserving robustness to hyperparameters. The findings suggest that, in o.o.d. settings, enriching representations rather than sparsifying them yields meaningful gains, with practical implications for deploying large foundational models in diverse deployment contexts.

Abstract

It is impossible today to pretend that the practice of machine learning is always compatible with the idea that training and testing data follow the same distribution. Several authors have recently used ensemble techniques to show how scenarios involving multiple data distributions are best served by representations that are both richer than those obtained by regularizing for the best in-distribution performance, and richer than those obtained under the influence of the implicit sparsity bias of common stochastic gradient procedures. This contribution investigates the use of very high dropout rates instead of ensembles to obtain such rich representations. Although training a deep network from scratch using such dropout rates is virtually impossible, fine-tuning a large pre-trained model under such conditions is not only possible but also achieves out-of-distribution performances that exceed those of both ensembles and weight averaging methods such as model soups. This result has practical significance because the importance of the fine-tuning scenario has considerably grown in recent years. This result also provides interesting insights on the nature of rich representations and on the intrinsically linear nature of fine-tuning a large network using a comparatively small dataset.

Fine-tuning with Very Large Dropout

TL;DR

This paper tackles the problem of out-of-distribution generalization when fine-tuning pre-trained models. It introduces a simple yet effective approach: applying very large dropout (around 90%) to the penultimate layer during fine-tuning to force the model to rely on a richer, more redundant set of features. Empirical results across multiple DomainBed tasks show that very-large dropout outperforms ensemble and weight-averaging baselines, and remains compatible with other fine-tuning tricks, while preserving robustness to hyperparameters. The findings suggest that, in o.o.d. settings, enriching representations rather than sparsifying them yields meaningful gains, with practical implications for deploying large foundational models in diverse deployment contexts.

Abstract

It is impossible today to pretend that the practice of machine learning is always compatible with the idea that training and testing data follow the same distribution. Several authors have recently used ensemble techniques to show how scenarios involving multiple data distributions are best served by representations that are both richer than those obtained by regularizing for the best in-distribution performance, and richer than those obtained under the influence of the implicit sparsity bias of common stochastic gradient procedures. This contribution investigates the use of very high dropout rates instead of ensembles to obtain such rich representations. Although training a deep network from scratch using such dropout rates is virtually impossible, fine-tuning a large pre-trained model under such conditions is not only possible but also achieves out-of-distribution performances that exceed those of both ensembles and weight averaging methods such as model soups. This result has practical significance because the importance of the fine-tuning scenario has considerably grown in recent years. This result also provides interesting insights on the nature of rich representations and on the intrinsically linear nature of fine-tuning a large network using a comparatively small dataset.
Paper Structure (33 sections, 2 equations, 2 figures, 5 tables)

This paper contains 33 sections, 2 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Constructing versatile representations
  4. Fine-tuning as a near-linear process
  5. Fine-tuning with very large dropout
  6. Fine-tuning and dropout
  7. The three-distributions setup
  8. Examples
  9. Method
  10. Experiments
  11. Dataset
  12. Models
  13. Pre-training
  14. Baselines
  15. Very large dropout yields better o.o.d. performance
  16. ...and 18 more sections

Figures (2)

  • Figure 1: o.o.d. performance comparison between very large dropout, ensembles, and weight averaging methods on four DomainBed tasks. erm results were obtained using plain fine-tuning with different hyperparameters. Weight averaging results either average the model weights collected every 300 iterations along each fine-tuning trajectory or the final model weights of all fine-tuning trajectories as in rame2022diverse. Ensemble results average instead the model outputs. Finally, large dropout results were obtained like the erm results but using a 90% dropout rate on the penultimate layer. Each box summarizes the results obtained with different hyper-parameters combinations.
  • Figure 2: Comparison of dropout rates when training a ResNet50 network from scratch on the vlcs dataset. The optimal dropout rate falls to about zero. Dropout rates greater than 50% negatively impact both the i.i.d. and the o.o.d. performances.