Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, Jianfeng Gao

TL;DR

Elevater introduces a public benchmark and toolkit to rigorously evaluate task-level transfer of language-augmented visual models across diverse, knowledge-enriched downstream tasks. It combines ICinW and ODinW datasets with external knowledge sources, an automatic hyper-parameter tuning pipeline, and metrics for zero-shot, few-shot, and full-shot as well as linear probing and full fine-tuning. The study demonstrates that language-initialized adaptation consistently improves transfer, that language-augmented models outperform language-free baselines in low-data regimes, and that external knowledge further boosts performance, especially in zero/few-shot settings. By providing a scalable, fair, and knowledge-aware evaluation framework, Elevater aims to advance research on task-level visual transfer under CVinW settings with practical implications for real-world deployment.

Abstract

Learning visual representations from natural language supervision has recently shown great promise in a number of pioneering works. In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks. However, it remains challenging to evaluate the transferablity of these models due to the lack of easy-to-use evaluation toolkits and public benchmarks. To tackle this, we build ELEVATER (Evaluation of Language-augmented Visual Task-level Transfer), the first benchmark and toolkit for evaluating(pre-trained) language-augmented visual models. ELEVATER is composed of three components. (i) Datasets. As downstream evaluation suites, it consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. (ii) Toolkit. An automatic hyper-parameter tuning toolkit is developed to facilitate model evaluation on downstream tasks. (iii) Metrics. A variety of evaluation metrics are used to measure sample-efficiency (zero-shot and few-shot) and parameter-efficiency (linear probing and full model fine-tuning). ELEVATER is a platform for Computer Vision in the Wild (CVinW), and is publicly released at at https://computer-vision-in-the-wild.github.io/ELEVATER/

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

TL;DR

Elevater introduces a public benchmark and toolkit to rigorously evaluate task-level transfer of language-augmented visual models across diverse, knowledge-enriched downstream tasks. It combines ICinW and ODinW datasets with external knowledge sources, an automatic hyper-parameter tuning pipeline, and metrics for zero-shot, few-shot, and full-shot as well as linear probing and full fine-tuning. The study demonstrates that language-initialized adaptation consistently improves transfer, that language-augmented models outperform language-free baselines in low-data regimes, and that external knowledge further boosts performance, especially in zero/few-shot settings. By providing a scalable, fair, and knowledge-aware evaluation framework, Elevater aims to advance research on task-level visual transfer under CVinW settings with practical implications for real-world deployment.

Abstract

Learning visual representations from natural language supervision has recently shown great promise in a number of pioneering works. In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks. However, it remains challenging to evaluate the transferablity of these models due to the lack of easy-to-use evaluation toolkits and public benchmarks. To tackle this, we build ELEVATER (Evaluation of Language-augmented Visual Task-level Transfer), the first benchmark and toolkit for evaluating(pre-trained) language-augmented visual models. ELEVATER is composed of three components. (i) Datasets. As downstream evaluation suites, it consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. (ii) Toolkit. An automatic hyper-parameter tuning toolkit is developed to facilitate model evaluation on downstream tasks. (iii) Metrics. A variety of evaluation metrics are used to measure sample-efficiency (zero-shot and few-shot) and parameter-efficiency (linear probing and full model fine-tuning). ELEVATER is a platform for Computer Vision in the Wild (CVinW), and is publicly released at at https://computer-vision-in-the-wild.github.io/ELEVATER/
Paper Structure (59 sections, 1 equation, 10 figures, 17 tables)

This paper contains 59 sections, 1 equation, 10 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work: From Class-level to Task-level Transfer
  3. Benchmarks
  4. A Suite of Datasets with Language/Knowledge Augmentations
  5. Pre-trained Models for Transfer Learning
  6. Industry Track and Academic Track.
  7. Pre-trained Models.
  8. Evaluation Settings: Efficiency Considerations
  9. Sample-efficiency: Zero-, Few-, and Full-shot.
  10. Parameter-efficiency: Linear Probing vs Full Model Fine-tuning.
  11. Toolkits
  12. Language-free Visual Models.
  13. Language-augmented Visual Models.
  14. Discussion.
  15. Empirical Results and Findings
  16. ...and 44 more sections

Figures (10)

  • Figure 1: The illustrative pipeline to use Elevater to evaluate a model checkpoint.
  • Figure 2: Illustration of CVinW in comparison with close-set, open-set and domain shift.
  • Figure 3: Illustration of our benchmark. Left: Image classification, Right: Object detection.
  • Figure 4: The model adaptation cost chart.
  • Figure 5: Illustrative comparison of different model evaluation and adaption methods.
  • ...and 5 more figures