Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

TL;DR

The paper addresses how to leverage a reference model to improve generalization and data efficiency in training a target model. It introduces DRRho risk minimization, a DRO-based framework that uses a shifted loss to stabilize learning and reduce loss variance, and derives generalization bounds to explain its benefits. It then specializes the framework to CLIP, producing DRRho-CLIP, and demonstrates, through extensive experiments, better data efficiency, stronger baselines, and a superior scaling law compared to standard CLIP. The work provides a theory-grounded justification for model steering and shows practical gains in large-scale vision-language learning, with implications for more data-efficient training and optimized resource use in foundation-model development.

Abstract

This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting, named $\textbf{model steering}$. While ad-hoc methods have been used in various contexts, including the training of large foundation models, its underlying principles remain insufficiently understood, leading to sub-optimal performance. In this work, we propose a theory-driven framework for model steering called $\textbf{DRRho risk minimization}$, which is rooted in Distributionally Robust Optimization (DRO). Through a generalization analysis, we provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model. To the best of our knowledge, this is the first time such theoretical insights are provided for the new learning paradigm, which significantly enhance our understanding and practice of model steering. Building on these insights and the connection between contrastive learning and DRO, we introduce a novel method for Contrastive Language-Image Pretraining (CLIP) with a reference model, termed DRRho-CLIP. Extensive experiments validate the theoretical insights, reveal a superior scaling law compared to CLIP without a reference model, and demonstrate its strength over existing heuristic approaches.

Figures (6)

• Figure 1: Comparison between a target model (ViT-B/16) trained by the proposed DRRho-CLIP and the reference model it leverages. OpenAI CLIP (ViT-B/32) was trained on a private 400M dataset with 12.8B samples seen and 32768 batch size. DRRho-CLIP model was trained on DFN-192M with 1.28B samples seen and 5120 batch size, and using OpenAI CLIP as a reference model .
• Figure 2: Scaling performance of OpenCLIP cherti2023reproducible and the proposed DRRho-CLIP, which uses the OpenAI CLIP model radford2021learning as the reference model. We conduct experiments of the two methods under different settings to fit scaling laws, as shown in the bottom left corner (c.f. \ref{['sec:experiments']} for more detail).
• Figure 3: Performance curves of FastCLIP and DRRho-CLIP with different target and reference models (with each column representing one combination). Top row: ImageNet Top-1 accuracy, bottom row: Datacomp average performance.
• Figure 4: Zero-shot Top 1 Accuracy on ImageNet-1K of different models. DFN model was trained on DFN-192M dataset with 1.28B samples seen with batch size 8192 fang2024data, DRRho-CLIP model was trained under the same setting with batch size 5120, and using OpenAI CLIP as a reference model.
• Figure 5: ImageNet Top 1 accuracy curves of DRRho-CLIP with fixed and learnable temperature on different target models and reference models.

Theorems & Definitions (11)

• Theorem 3.1
• Theorem 4.1
• Corollary 4.2
• Corollary 4.3
• Theorem 1.1: Theorem 3 in duchi2016variance
• Corollary 1.2: Corollary 3.1 in duchi2016variance
• Theorem 1.3
• proof
• proof : Proof of \ref{['thm:ref']}
• proof : Proof of \ref{['cor:ref']}