Multimodal Data Curation via Object Detection and Filter Ensembles

TL;DR

This work tackles data quality in web-sourced multimodal datasets by proposing a two-stage curation framework that first creates granular, detector-informed filters from zero-shot object detection (Grounding DINO) and CLIP cues, then fuses these signals with a weak-supervision Ising-model ensemble (via Snorkel) to select a high-quality subset for training. The approach yields consistent gains on DataComp benchmarks: a $4\%$ improvement in the small-scale track and a $4.2\%$ improvement in the medium-scale track over strong baselines, with final metrics including ImageNet accuracy of $0.305$ and average performance of $0.342$ in the medium scale. Key contributions include demonstrating the value of granular object-detection-derived filters, quantifying the benefits of weak supervision-based ensembling, and validating the framework's applicability across data scales. The work offers a practical, scalable data-curation pipeline that can improve the quality of large multimodal datasets without additional labeling effort, supporting more robust downstream vision-language models.

Abstract

We propose an approach for curating multimodal data that we used for our entry in the 2023 DataComp competition filtering track. Our technique combines object detection and weak supervision-based ensembling. In the first of two steps in our approach, we employ an out-of-the-box zero-shot object detection model to extract granular information and produce a variety of filter designs. In the second step, we employ weak supervision to ensemble filtering rules. This approach results in a 4% performance improvement when compared to the best-performing baseline, producing the top-ranking position in the small scale track at the time of writing. Furthermore, in the medium scale track, we achieve a noteworthy 4.2% improvement over the baseline by simply ensembling existing baselines with weak supervision.

Figures (4)

• Figure 1: Overall workflow of the data curation framework. Each data point in the raw data pool is passed to individual filters, which can be designed by human heuristics, pre-computed CLIP scores, or inference results from other off-the-shelf models. We employ Grounding DINO, a zero-shot object detection model, to identify objects mentioned in the image caption. After each designed filter processes, we ensemble filtering results and curate the final refined data pool.
• Figure 2: We showcase various image samples with their caption and CLIP score in the small scale dataset and annotate recognized objects through Grounding DINO. Nearly 38% of the images do not have any identified objects, while 18% of them have multiple detected objects. Additionally, around 3% of the images have tiny detected objects. These results offer rich information, which can be used and combined with heuristics to design additional filtering rules.
• Figure 3: Correlation across designed filters.
• Figure 4: Estimated accuracy across designed filters.