Generating Enhanced Negatives for Training Language-Based Object Detectors

TL;DR

This work tackles the challenge of training language-based object detectors under an open vocabulary by generating high-quality negatives automatically. It combines large language models to produce semantically related but contradictory negative texts and text-to-image diffusion models to create corresponding negative images, followed by targeted filtering to reduce noise. By integrating these negatives into detectors like GLIP and FIBER, the authors demonstrate systematic performance gains on OmniLabel and D$^3$xie benchmarks, highlighting the value of diverse, model-generated negatives for discriminative training. The approach provides practical recipes for leveraging generative models in detection pipelines and analyzes the contribution of text and image negatives to the improved performance.

Abstract

The recent progress in language-based open-vocabulary object detection can be largely attributed to finding better ways of leveraging large-scale data with free-form text annotations. Training such models with a discriminative objective function has proven successful, but requires good positive and negative samples. However, the free-form nature and the open vocabulary of object descriptions make the space of negatives extremely large. Prior works randomly sample negatives or use rule-based techniques to build them. In contrast, we propose to leverage the vast knowledge built into modern generative models to automatically build negatives that are more relevant to the original data. Specifically, we use large-language-models to generate negative text descriptions, and text-to-image diffusion models to also generate corresponding negative images. Our experimental analysis confirms the relevance of the generated negative data, and its use in language-based detectors improves performance on two complex benchmarks. Code is available at \url{https://github.com/xiaofeng94/Gen-Enhanced-Negs}.

Figures (17)

• Figure 1: The key contribution of our work is to leverage large-language-models and text-to-image diffusion models to automatically generate negative object descriptions and images for training language-based object detectors. In contrast to prior work, our generated negatives are more relevant to the original data and provide a better training signal for detectors.
• Figure 2: In language-based object detection, a detector receives as input an image and a (variable-length) list of free-form text descriptions of objects. For each description, the model predicts bounding boxes for objects that match the description.
• Figure 3: Overview of using LLMs openai2022gpt-3.5touvron2023llama and text-to-image diffusion models rombach2022highli2023gligen to generate negative images.
• Figure 4: Illustration of the grounding loss used for training. Predictions that are matched with ground truth receive a positive signal from the associated text (tall rectangles). All other words receive a negative signal (short rectangles). The top left quarter shows the original loss used in dou2022coarse_fiberli2022grounded. The other three quarters are related to our proposed generated negative data and provide additional training signals.
• Figure 5: Percentage of negative texts with the numbers of words. Four negative generation methods are compared.