Webly Supervised Concept Expansion for General Purpose Vision Models

TL;DR

Web10k enables webly supervised concept expansion for GPVs by learning concepts from the web while learning skills from supervised data. The authors introduce GPV-2, a box-capable, unified vision-language architecture with a shared decoder and Language-Based Localization that supports a wide range of tasks, from classification to HOI detection, and demonstrate strong cross-task transfer. Across Coco-sce, DCE, and Web10k benchmarks, web data consistently improves performance and GPV-2 outperforms prior GPV architectures, including a demonstration of rapid adaptation to new concepts (e.g., COVID-related terms). The work provides a scalable, cost-effective path to expanding GPV concept vocabularies and capabilities, with public data, benchmarks, and code to facilitate broader adoption and evaluation.

Abstract

General Purpose Vision (GPV) systems are models that are designed to solve a wide array of visual tasks without requiring architectural changes. Today, GPVs primarily learn both skills and concepts from large fully supervised datasets. Scaling GPVs to tens of thousands of concepts by acquiring data to learn each concept for every skill quickly becomes prohibitive. This work presents an effective and inexpensive alternative: learn skills from supervised datasets, learn concepts from web image search, and leverage a key characteristic of GPVs: the ability to transfer visual knowledge across skills. We use a dataset of 1M+ images spanning 10k+ visual concepts to demonstrate webly-supervised concept expansion for two existing GPVs (GPV-1 and VL-T5) on 3 benchmarks: 5 COCO-based datasets (80 primary concepts), a newly curated series of 5 datasets based on the OpenImages and VisualGenome repositories (~500 concepts), and the Web-derived dataset (10k+ concepts). We also propose a new architecture, GPV-2 that supports a variety of tasks -- from vision tasks like classification and localization to vision+language tasks like QA and captioning, to more niche ones like human-object interaction detection. GPV-2 benefits hugely from web data and outperforms GPV-1 and VL-T5 across these benchmarks. Our data, code, and web demo are available at https://prior.allenai.org/projects/gpv2.

Figures (9)

• Figure 1: Learning concepts from the web with GPV-2. We demonstrate webly-supervised concept expansion on two existing GPV architectures (GPV-1 and VL-T5) as well as our proposed GPV-2 architecture. In addition to outperforming previous architectures, GPV-2 expands the inputs to contain bounding boxes which enables support for niche tasks like Human-Object Interaction detection with multi-step inference without any architectural modifications.
• Figure 1: Qualitative examples for GPV-2. Examples are from DCE val, except for the last image in each row, which comes from Coco val. GPV-2 is able to use concepts that do not appear in the Coco training data across all five skills.
• Figure 2: Concept diversity in WEB10K.Left: Besides 10k nouns, Web10k provides dense coverage of feasible adj-noun and verb-noun combinations to enable learning of fine-grained differences in object appearance due to attributes. Right: TSNE Maaten2008tsne plot of Phrase-BERT Wang2021PhraseBERT embeddings of Web10k nouns with bubble size indicating frequency (capped at 1000) in CC, a common large-scale pretraining dataset. Web10k nouns cover a wide range of concept groups identified using WordNet and include many concepts which are infrequent/absent in CC.
• Figure 2: Qualitative Examples: GPV-2 on DCE, with and without training on WEB10K. The use of Web10k allows GPV-2 to understand more concepts across all skills, especially for rare concepts such as "red panda" (captioning upper right).
• Figure 3: Left: GPV-2 architecture. Right: I/O for 5 skills in Coco and DCE.