Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models

Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, Yunhe Wang

TL;DR

This work tackles the problem of enabling superhuman-style performance in vision models by enabling weak-to-strong generalization (WSG), where a weaker teacher guides a stronger vision foundation model. The authors introduce Adaptive Confidence Distillation (AdaptConf), a confidence-weighted loss that dynamically balances learning from a weak teacher with the strong model's own predictions via a per-sample weight $\beta(x)$ computed from cross-entropy discrepancies. They validate the approach across image classification, few-shot, transfer, and noisy-label settings, showing that AdaptConf surpasses strong-to-strong distillation and full-dataset fine-tuning baselines, with robust ablations and analysis. The results suggest a practical, scalable path to improve vision foundation models by leveraging imperfect human-like guidance, accompanied by public code for reproduction.

Abstract

Recent advancements in large language models have sparked interest in their extraordinary and near-superhuman capabilities, leading researchers to explore methods for evaluating and optimizing these abilities, which is called superalignment. In this context, our paper delves into the realm of vision foundation models, focusing on the concept of weak-to-strong generalization, which involves using a weaker model to supervise a stronger one, aiming to enhance the latter's capabilities beyond the former's limits. We introduce a novel and adaptively adjustable loss function for weak-to-strong supervision. Our comprehensive experiments span various scenarios, including few-shot learning, transfer learning, noisy label learning, and common knowledge distillation settings. The results are striking: our approach not only exceeds the performance benchmarks set by strong-to-strong generalization but also surpasses the outcomes of fine-tuning strong models with whole datasets. This compelling evidence underscores the significant potential of weak-to-strong generalization, showcasing its capability to substantially elevate the performance of vision foundation models. The code is available at https://github.com/ggjy/vision_weak_to_strong.

Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models

TL;DR

This work tackles the problem of enabling superhuman-style performance in vision models by enabling weak-to-strong generalization (WSG), where a weaker teacher guides a stronger vision foundation model. The authors introduce Adaptive Confidence Distillation (AdaptConf), a confidence-weighted loss that dynamically balances learning from a weak teacher with the strong model's own predictions via a per-sample weight computed from cross-entropy discrepancies. They validate the approach across image classification, few-shot, transfer, and noisy-label settings, showing that AdaptConf surpasses strong-to-strong distillation and full-dataset fine-tuning baselines, with robust ablations and analysis. The results suggest a practical, scalable path to improve vision foundation models by leveraging imperfect human-like guidance, accompanied by public code for reproduction.

Abstract

Recent advancements in large language models have sparked interest in their extraordinary and near-superhuman capabilities, leading researchers to explore methods for evaluating and optimizing these abilities, which is called superalignment. In this context, our paper delves into the realm of vision foundation models, focusing on the concept of weak-to-strong generalization, which involves using a weaker model to supervise a stronger one, aiming to enhance the latter's capabilities beyond the former's limits. We introduce a novel and adaptively adjustable loss function for weak-to-strong supervision. Our comprehensive experiments span various scenarios, including few-shot learning, transfer learning, noisy label learning, and common knowledge distillation settings. The results are striking: our approach not only exceeds the performance benchmarks set by strong-to-strong generalization but also surpasses the outcomes of fine-tuning strong models with whole datasets. This compelling evidence underscores the significant potential of weak-to-strong generalization, showcasing its capability to substantially elevate the performance of vision foundation models. The code is available at https://github.com/ggjy/vision_weak_to_strong.
Paper Structure (19 sections, 2 equations, 3 figures, 7 tables)

This paper contains 19 sections, 2 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Vision Superalignment
  4. Vision Foundation Models
  5. Adaptive Confidence Distillation
  6. Experiment
  7. Tasks
  8. Main Results
  9. Image Classification.
  10. Few-shot learning
  11. Transfer learning
  12. Learning with noisy labels
  13. Ablation Study
  14. Conclusion
  15. Implementation details.
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Our proposed AdaptConf achieves the best performance on a broad range of tasks compared with other knowledge distillation based methods. The corresponding values are calculated by averaging results on eack task. CLS-CIFAR-S: Table \ref{['tab:cifar_cls_same']}, CLS-CIFAR-D: Table \ref{['tab:cifar_cls_different_gt']}, CLS-ImageNet-S: Table \ref{['tab:imagenet_cls']}, CLS-ImageNet-D: Table \ref{['tab:imagenet_cls']}, TL-ImageNet: Table \ref{['tab:imagenet_transfer']}, TL-iNat: Table \ref{['tab:inat_transfer']}, FSL-miniImageNet: Table \ref{['tab:meta_base-class']}, LNL-CIFAR: Table \ref{['tab:lnl']}.
  • Figure 2: Ablation study examining the impact of hyper-parameter variation on confidence distillation results. The parameter $\alpha$ for AugConf is adjusted across a range from 0.1 to 0.9, while the temperature $T$ for AdaptConf is scaled from 0.1 to 8.
  • Figure 3: Quantitative analysis about the value of $\beta$(x) in Eq. \ref{['eq:adaptconf']} on the CIFAR-100 dataset. The evaluation is based on the ShuffleNetV1-ResNet32x4 teacher-student architecture pair.