Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters

Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, Heng Wang

TL;DR

This work tackles the data quality bottleneck in large-scale image-text datasets by introducing a fine-tuned Multimodal Language Model (MLM) data filter that surpasses CLIPScore. It defines four complementary evaluation metrics—Image-Text Matching (ITM), Object Detail Fulfillment (ODF), Caption Text Quality (CTQ), and Semantic Understanding (SU)—and trains MLMs to score image-text pairs via multimodal instruction tuning data constructed with teacher models (GPT-4 and GPT-4V). The authors demonstrate, through DataComp-based experiments and pre-training of CLIP and BLIP-2 models, that MLM-filtered data improves downstream tasks and correlates more strongly with human quality judgments than CLIPScore. The approach scales to different base models and offers a practical drop-in replacement for CLIPScore, with ablations validating design choices and efficiency optimizations. Altogether, the results highlight the practical impact of MLM-based data filtering for robust, high-quality vision-language pre-training.

Abstract

We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.

Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters

TL;DR

This work tackles the data quality bottleneck in large-scale image-text datasets by introducing a fine-tuned Multimodal Language Model (MLM) data filter that surpasses CLIPScore. It defines four complementary evaluation metrics—Image-Text Matching (ITM), Object Detail Fulfillment (ODF), Caption Text Quality (CTQ), and Semantic Understanding (SU)—and trains MLMs to score image-text pairs via multimodal instruction tuning data constructed with teacher models (GPT-4 and GPT-4V). The authors demonstrate, through DataComp-based experiments and pre-training of CLIP and BLIP-2 models, that MLM-filtered data improves downstream tasks and correlates more strongly with human quality judgments than CLIPScore. The approach scales to different base models and offers a practical drop-in replacement for CLIPScore, with ablations validating design choices and efficiency optimizations. Altogether, the results highlight the practical impact of MLM-based data filtering for robust, high-quality vision-language pre-training.

Abstract

We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.
Paper Structure (19 sections, 4 figures, 9 tables)

This paper contains 19 sections, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fine-Tuned Multimodal Language Models as Data Filters
  4. Overview
  5. Constructing Multimodal Instruction Tuning Data for Scoring Tasks
  6. Instruction-Tuning on Multimodal Language Models
  7. Creating Optimal MLM Data Filters
  8. Ablation Results.
  9. Experiments
  10. CLIP Pre-Training on DataComp Medium and Large Scales
  11. BLIP2 Pre-Training
  12. Correlation with Human Scoring
  13. Analysis
  14. Conclusion
  15. Prompt Construction
  16. ...and 4 more sections

Figures (4)

  • Figure 1: CLIPScore fails in differentiating the fine-grained object-level image-text alignment, while the image-text matching score generated by MLM Filter significantly captures such alignment.
  • Figure 2: Illustration of the pipeline of fine-tuning MLM Filter and employing it for data filtering.
  • Figure 3: (a) image text matching score distribution of initial 10k instructions using GPT-4V on CC12M; (b) image text matching score distribution of final 1k instructions uniformly sampled from 10 buckets.
  • Figure 4: Effects of fraction of images selected for training CLIP.