Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detection

TL;DR

The paper addresses the challenge of compositional understanding in language-based object detection by introducing Weak-to-Strong Compositional Learning (WSCL), which leverages generative foundation models to automatically create dense synthetic <image, description, bounding box> triplets. A two-stage approach combines dense triplet generation (via LLMs and diffusion, with weak-to-strong pseudo bounding boxes) and compositional contrastive learning that enforces description-awareness and textural-structural-awareness, while mitigating domain bias. Empirical results on OmniLabel and D3 show significant gains over strong baselines (e.g., +5.0 AP on OmniLabel, +6.9 AP on D3) and competitive improvements when paired with DesCo, demonstrating model-agnostic applicability and practical impact for open-vocabulary object detection. This work highlights how distilling compositional reasoning from generative models into VL detectors can improve detection of objects described by complex attributes and relations, with potential to reduce annotation costs. The method advances open-vocabulary detection by enhancing the model’s ability to reason about detailed descriptions in real-world scenes.

Abstract

Vision-language (VL) models often exhibit a limited understanding of complex expressions of visual objects (e.g., attributes, shapes, and their relations), given complex and diverse language queries. Traditional approaches attempt to improve VL models using hard negative synthetic text, but their effectiveness is limited. In this paper, we harness the exceptional compositional understanding capabilities of generative foundational models. We introduce a novel method for structured synthetic data generation aimed at enhancing the compositional understanding of VL models in language-based object detection. Our framework generates densely paired positive and negative triplets (image, text descriptions, and bounding boxes) in both image and text domains. By leveraging these synthetic triplets, we transform 'weaker' VL models into 'stronger' models in terms of compositional understanding, a process we call "Weak-to-Strong Compositional Learning" (WSCL). To achieve this, we propose a new compositional contrastive learning formulation that discovers semantics and structures in complex descriptions from synthetic triplets. As a result, VL models trained with our synthetic data generation exhibit a significant performance boost in the Omnilabel benchmark by up to +5AP and the D3 benchmark by +6.9AP upon existing baselines.

Figures (6)

• Figure 1: (a-b) While previous models lack a compositional understanding of the given language query and localize wrong objects, resulting in higher scores for the wrong objects (i.e., the middle woman and black tops) than the actual object, our method successfully localizes only correct objects corresponding to the query. (c) Previous VL methods apply (e.g., svlcdesco) augmentation exclusively in the text domain. (d) The proposed method produces comprehensive synthetic triplets comprising <image, object description, bounding box>, incorporating compositional contrastive learning to improve the model’s understanding of composition.
• Figure 2: Overview of our method. Our method consists of generating dense synthetic image-text paired triplets with generative models and creating bounding boxes. Finally, we introduce compositional conservative learning with our generated triplets which enhances the model's compositional ability in language-based object detection.
• Figure 3: (a) Qualitative examples of generated synthetic images and descriptions. (b) Comparison between grounding-based labeling and our weak-to-strong labeling. (c) Illustration of our weak-to-strong labeling, where we decompose the complex task into easy tasks. The bounding boxes collected from each task are combined to create strong compositional labels that train a strong detector.
• Figure 4: Illustration of our compositional contrastive learning. (a) Intra-class negatives from other images of the same class and structural positives are introduced to learn the context of descriptions. (b) We associate the sentence-level positive (i.e., the entire description sentence) with the pseudo bounding box of the "an avocado" while differentiating the structure negative (i.e., the noun phrase "a cutting board") from the pseudo bounding box of the "a cutting board".
• Figure 5: Qualitative comparisons on OmniLabel omnilabel benchmark. We can observe clear improvements in compositional understanding against GLIP glip and Desco-GLIP desco.