Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection

Arushi Rai, Adriana Kovashka

TL;DR

This work tackles the pervasive problem of label noise in large vision–language data used for weakly supervised object detection. It introduces VEIL, a transformer-based label-vetting model that uses caption context to predict whether extracted labels are visually present, trained with pseudo-ground-truth from an image-recognition ensemble. To support this, the authors construct the Caption Label Noise (CLaN) dataset to analyze visual and linguistic cues behind absent labels and demonstrate VEIL's superiority over nine baselines, including CLIP-based filters, across multiple in-the-wild datasets and in cross-dataset settings. Empirically, vetting with VEIL yields significant WSOD gains, with notable improvements on VOC-07 and COCO-14, and enables effective mixing of clean and noisy supervision with scalable performance gains. The work offers a practical pathway to leverage abundant in-the-wild captions for open-vocabulary WSOD while mitigating detrimental label noise.

Abstract

The use of large-scale vision-language datasets is limited for object detection due to the negative impact of label noise on localization. Prior methods have shown how such large-scale datasets can be used for pretraining, which can provide initial signal for localization, but is insufficient without clean bounding-box data for at least some categories. We propose a technique to "vet" labels extracted from noisy captions, and use them for weakly-supervised object detection (WSOD), without any bounding boxes. We analyze and annotate the types of label noise in captions in our Caption Label Noise dataset, and train a classifier that predicts if an extracted label is actually present in the image or not. Our classifier generalizes across dataset boundaries and across categories. We compare the classifier to nine baselines on five datasets, and demonstrate that it can improve WSOD without label vetting by 30% (31.2 to 40.5 mAP when evaluated on PASCAL VOC). See dataset at: https://github.com/arushirai1/CLaNDataset.

VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection

TL;DR

This work tackles the pervasive problem of label noise in large vision–language data used for weakly supervised object detection. It introduces VEIL, a transformer-based label-vetting model that uses caption context to predict whether extracted labels are visually present, trained with pseudo-ground-truth from an image-recognition ensemble. To support this, the authors construct the Caption Label Noise (CLaN) dataset to analyze visual and linguistic cues behind absent labels and demonstrate VEIL's superiority over nine baselines, including CLIP-based filters, across multiple in-the-wild datasets and in cross-dataset settings. Empirically, vetting with VEIL yields significant WSOD gains, with notable improvements on VOC-07 and COCO-14, and enables effective mixing of clean and noisy supervision with scalable performance gains. The work offers a practical pathway to leverage abundant in-the-wild captions for open-vocabulary WSOD while mitigating detrimental label noise.

Abstract

The use of large-scale vision-language datasets is limited for object detection due to the negative impact of label noise on localization. Prior methods have shown how such large-scale datasets can be used for pretraining, which can provide initial signal for localization, but is insufficient without clean bounding-box data for at least some categories. We propose a technique to "vet" labels extracted from noisy captions, and use them for weakly-supervised object detection (WSOD), without any bounding boxes. We analyze and annotate the types of label noise in captions in our Caption Label Noise dataset, and train a classifier that predicts if an extracted label is actually present in the image or not. Our classifier generalizes across dataset boundaries and across categories. We compare the classifier to nine baselines on five datasets, and demonstrate that it can improve WSOD without label vetting by 30% (31.2 to 40.5 mAP when evaluated on PASCAL VOC). See dataset at: https://github.com/arushirai1/CLaNDataset.
Paper Structure (18 sections, 2 equations, 4 figures, 15 tables)

This paper contains 18 sections, 2 equations, 4 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Label Noise Analysis and Dataset
  4. Method
  5. Experiments
  6. Experiment Details
  7. Methods Compared
  8. Extracted Label Vetting Evaluation
  9. Impact on Weakly-Sup. Object Detection
  10. Conclusion
  11. Appendix
  12. Quality of Pretrained Image Recognition Ensemble
  13. Vetting Dataset Details
  14. Vetting Precision/Recall
  15. Cross-Dataset Ablations
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Examples of noisy extracted labels (underlined ) from our Caption Label Noise dataset. We categorize types of similar context present instead of the underlined object, as well as types of visual defects and linguistic indicators that are useful for detecting noise.
  • Figure 2: VEIL architecture. In this example, only "dog" is an extracted label and it fails the vetting process. The masking layer masks visual presence predictions for text tokens not corresponding to an extracted label.
  • Figure 3: Qualitative examples of extracted labels after vetting on RedCaps-Test. These are additional completely absent VAEL examples from CLaN with their linguistic indicators and similar context annotations, and only VEIL-based methods are able to overcome these three noise types.
  • Figure 4: Detections (blue bounding box) from WSOD models trained with various vetting methods (top row) indicate that training with either VEIL-based vetting method (two rightmost columns) leads to similar detection capability on VOC-07 Everingham2010ThePV. The categories shown by row (from top to bottom) are: horse, car, boat, bicycle, chair.