ClipGrader: Leveraging Vision-Language Models for Robust Label Quality Assessment in Object Detection

TL;DR

ClipGrader introduces a CLIP-based framework that reframes bounding-box annotation quality as a vision-language grading task, enabling automatic assessment of both label correctness and spatial precision. By constructing magenta-box crops and a set of $2n+1$ prompts, ClipGrader learns via image-text contrastive losses to distinguish good, bad, and background boxes, achieving high accuracy on COCO and strong generalization to LVIS with limited data. The approach demonstrates data efficiency, zero-shot limitations, and notable utility when integrated into semi-supervised object detection (SSOD) as CLIP-Teacher, which filters pseudo-labels to improve detector performance. The work highlights the potential of AI-assisted data curation for large-scale detection datasets and outlines pathways for refining datasets, extending to other tasks, and leveraging larger vision-language models for even stronger guidance in annotation quality control.

Abstract

High-quality annotations are essential for object detection models, but ensuring label accuracy - especially for bounding boxes - remains both challenging and costly. This paper introduces ClipGrader, a novel approach that leverages vision-language models to automatically assess the accuracy of bounding box annotations. By adapting CLIP (Contrastive Language-Image Pre-training) to evaluate both class label correctness and spatial precision of bounding box, ClipGrader offers an effective solution for grading object detection labels. Tested on modified object detection datasets with artificially disturbed bounding boxes, ClipGrader achieves 91% accuracy on COCO with a 1.8% false positive rate. Moreover, it maintains 87% accuracy with a 2.1% false positive rate when trained on just 10% of the COCO data. ClipGrader also scales effectively to larger datasets such as LVIS, achieving 79% accuracy across 1,203 classes. Our experiments demonstrate ClipGrader's ability to identify errors in existing COCO annotations, highlighting its potential for dataset refinement. When integrated into a semi-supervised object detection (SSOD) model, ClipGrader readily improves the pseudo label quality, helping achieve higher mAP (mean Average Precision) throughout the training process. ClipGrader thus provides a scalable AI-assisted tool for enhancing annotation quality control and verifying annotations in large-scale object detection datasets.

Figures (8)

• Figure 1: Examples illustrating the three types of generated input images and top model predictions. Each generated image crop contains one magenta bounding box to mark the target object's location.
• Figure 2: Accuracy vs. Epochs for different training dataset configurations
• Figure 3: Trade-off between mean recall of good labels and false acceptance of bad labels for different model configurations
• Figure 4: Accuracy vs. Epochs for different fine-tuning strategies and prompt engineering
• Figure 5: Clip-Teacher vs. Consistent-Teacher