Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRIT: General Robust Image Task Benchmark

Tanmay Gupta, Ryan Marten, Aniruddha Kembhavi, Derek Hoiem

TL;DR

GRIT introduces a unified benchmark to evaluate generality, robustness, and calibration of vision systems across seven diverse tasks, multiple data sources, and broad concepts. It implements restricted and unrestricted tracks, per-concept sampling, and distortion-based robustness tests to enable fair, cross-task comparisons and to assess transfer to novel data and concepts. The framework emphasizes calibration through confidence-based metrics and provides a suite of baselines to illustrate current limitations in generalization and reliability. By design, GRIT aims to accelerate development of general-purpose vision systems capable of multi-task learning and robust operation in open-world settings.

Abstract

Computer vision models excel at making predictions when the test distribution closely resembles the training distribution. Such models have yet to match the ability of biological vision to learn from multiple sources and generalize to new data sources and tasks. To facilitate the development and evaluation of more general vision systems, we introduce the General Robust Image Task (GRIT) benchmark. GRIT evaluates the performance, robustness, and calibration of a vision system across a variety of image prediction tasks, concepts, and data sources. The seven tasks in GRIT are selected to cover a range of visual skills: object categorization, object localization, referring expression grounding, visual question answering, segmentation, human keypoint detection, and surface normal estimation. GRIT is carefully designed to enable the evaluation of robustness under image perturbations, image source distribution shift, and concept distribution shift. By providing a unified platform for thorough assessment of skills and concepts learned by a vision model, we hope GRIT catalyzes the development of performant and robust general purpose vision systems.

GRIT: General Robust Image Task Benchmark

TL;DR

GRIT introduces a unified benchmark to evaluate generality, robustness, and calibration of vision systems across seven diverse tasks, multiple data sources, and broad concepts. It implements restricted and unrestricted tracks, per-concept sampling, and distortion-based robustness tests to enable fair, cross-task comparisons and to assess transfer to novel data and concepts. The framework emphasizes calibration through confidence-based metrics and provides a suite of baselines to illustrate current limitations in generalization and reliability. By design, GRIT aims to accelerate development of general-purpose vision systems capable of multi-task learning and robust operation in open-world settings.

Abstract

Computer vision models excel at making predictions when the test distribution closely resembles the training distribution. Such models have yet to match the ability of biological vision to learn from multiple sources and generalize to new data sources and tasks. To facilitate the development and evaluation of more general vision systems, we introduce the General Robust Image Task (GRIT) benchmark. GRIT evaluates the performance, robustness, and calibration of a vision system across a variety of image prediction tasks, concepts, and data sources. The seven tasks in GRIT are selected to cover a range of visual skills: object categorization, object localization, referring expression grounding, visual question answering, segmentation, human keypoint detection, and surface normal estimation. GRIT is carefully designed to enable the evaluation of robustness under image perturbations, image source distribution shift, and concept distribution shift. By providing a unified platform for thorough assessment of skills and concepts learned by a vision model, we hope GRIT catalyzes the development of performant and robust general purpose vision systems.
Paper Structure (13 sections, 2 equations, 3 figures, 13 tables)

This paper contains 13 sections, 2 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Dataset Design Principles
  3. Task Overview
  4. Challenge
  5. Concepts
  6. Metrics
  7. Robustness
  8. Knowledge and Calibration
  9. Recommended metrics
  10. Baselines
  11. Related Generalization Benchmarks
  12. Conclusion
  13. Acknowledgements

Figures (3)

  • Figure 1: A benchmark for more general vision systems. GRIT tests the generality, robustness, and calibration of a vision system across 7 vision and vision-language tasks, multiple data sources, and diverse concepts.
  • Figure 2: Inputs and ground truth task outputs for each of the 7 task in GRIT. For the categorization task, instead of an input query, we provide a list of categories to choose from.
  • Figure 3: Effect of per-concept sampling on VQAv2 dataset. In the plot, each point corresponds to a concept. With $N=50$, per-concept sampling generally selects more samples for concepts that would have been represented less than 50 times under random sampling while reduces the representation of concepts that appear more than 50 times during random sampling.