Scaling Laws for Data-Efficient Visual Transfer Learning

TL;DR

This work develops data-efficient scaling laws for visual transfer learning, addressing how performance scales when downstream data are limited and how distillation affects outcomes under such constraints. It introduces a distillation boundary theory and a joint power-law framework linking pretraining data, model size, and fine-tuning data, validated across multiple datasets and model scales from 2.5M to 38M parameters. The findings show a turning point: distillation provides clear benefits in data-scarce settings, but as pretraining data grows, non-distilled training can surpass distilled approaches, guiding resource allocation. By bridging the gap between large-scale pretraining and practical downstream adaptation, the paper offers actionable guidelines for selecting training strategies and predicting performance in data-limited vision tasks.

Abstract

Current scaling laws for visual AI models focus predominantly on large-scale pretraining, leaving a critical gap in understanding how performance scales for data-constrained downstream tasks. To address this limitation, this paper establishes the first practical framework for data-efficient scaling laws in visual transfer learning, addressing two fundamental questions: 1) How do scaling behaviors shift when downstream tasks operate with limited data? 2) What governs the efficacy of knowledge distillation under such constraints? Through systematic analysis of vision tasks across data regimes (1K-1M samples), we propose the distillation boundary theory, revealing a critical turning point in distillation efficiency: 1) Distillation superiority: In data-scarce conditions, distilled models significantly outperform their non-distillation counterparts, efficiently leveraging inherited knowledge to compensate for limited training samples. 2) Pre-training dominance: As pre-training data increases beyond a critical threshold, non-distilled models gradually surpass distilled versions, suggesting diminishing returns from knowledge inheritance when sufficient task-specific data becomes available. Empirical validation across various model scales (2.5M to 38M parameters) and data volumes demonstrate these performance inflection points, with error difference curves transitioning from positive to negative values at critical data thresholds, confirming our theoretical predictions. This work redefines scaling laws for data-limited regimes, bridging the knowledge gap between large-scale pretraining and practical downstream adaptation, addressing a critical barrier to understanding vision model scaling behaviors and optimizing computational resource allocation.

Figures (19)

• Figure 1: Visualization of the Distillation Boundary Theory. Empirical results reveal the existence of a critical pretraining data threshold: prior to this threshold, distillation enhances model generalization, while beyond it, the added constraints of distillation may hinder the model's capacity to fully benefit from abundant data, leading to a decline in performance.
• Figure 2: Scaling laws for pre-training data size on ImageNet100. Illustration of the impact of pre-training data volume ($Dp$) on error rate across different model sizes ($M$) on ImageNet100.
• Figure 3: Scaling laws for pre-training data size on ImageNet100. Illustration of the impact of pre-training data volume ($Dp$) on cross-entropy loss across different model sizes ($M$) on ImageNet100.
• Figure 5: Scaling laws for fine-tuning data size on ImageNet100. Illustration of the impact of fine-tuning data volume ($D_f$) on error rate across different model sizes ($M$) on ImageNet100.
• Figure 6: Scaling laws for fine-tuning data size on ImageNet100. Illustration of the impact of fine-tuning data volume ($D_f$) on cross-entropy loss across different model sizes ($M$) on ImageNet100.

Theorems & Definitions (5)

• Theorem 1: Distillation Efficacy Boundary
• Remark 1
• Lemma 1: Differential Error Function
• Corollary 1
• Remark 2