Train a Unified Multimodal Data Quality Classifier with Synthetic Data

TL;DR

UniFilter introduces a unified multimodal data quality classifier that scores both image-text captions and interleaved documents using a semi-synthetic data generation pipeline across four quality levels. By training an MLLM-based classifier on labeled sample-score pairs, UniFilter enables scalable, high-throughput data curation that surpasses CLIP-based and prior model-based filters. Empirical results show significant gains in zero-shot and few-shot multimodal reasoning when pre-training MLLMs on UniFilter-curated data, with further improvements after visual supervised fine-tuning. The work also provides public synthetic data, model checkpoints, and high-quality interleaved data subsets to support reproducibility and further development in multimodal data filtering.

Abstract

The Multimodal Large Language Models (MLLMs) are continually pre-trained on a mixture of image-text caption data and interleaved document data, while the high-quality data filtering towards image-text interleaved document data is under-explored. We propose to train an efficient MLLM as a Unified Mulitmodal Data Quality Classifier to Filter both high-quality image-text caption and interleaved data (UniFilter). To address the challenge of collecting diverse labeled multimodal data, we introduce a semi-synthetic approach that leverages readily available raw images and generates corresponding text across four quality levels. This method enables efficient creation of sample-score pairs for both caption and interleaved document data to train UniFilter. We apply UniFilter to curate high-quality caption data from DataComp caption dataset and interleaved data from the OBELICS image-text interleaved dataset. MLLMs pre-trained on the filtered data demonstrate significantly enhanced capabilities compared to those trained on baseline-filtered data, achieving stronger zero-shot reasoning and in-context learning capabilities. After visual supervised fine-tuning, these UniFilter-induced MLLMs achieve stronger performance on various benchmarks, highlighting the downstream benefits of high-quality multimodal pre-training. We release the synthetic training data used for training UniFilter, the UniFilter model checkpoints, and the high-quality interleaved document subset OBELICS-HQ, curated by UniFilter, to the community for reproduction and further development.

Figures (6)

• Figure 1: The pipeline of semi-synthetic data generation for image-text caption data and interleaved document data.
• Figure 2: The unified model architecture of UniFilter which uses an efficient MLLM to classify the quality scores of both image-text paired data (Left) and interleaved data (Right).
• Figure 3: The classification F1 versus inference speed of different MLLM architecture ablation configurations on the held validation data of quality classification task.
• Figure 4: The comparisons between UniFilter and DFN on each dataset in the 38 dataset evaluation suite of DataComp Meidum. The Green bars mean UniFilter beats DFN.
• Figure 5: A positive synthetic document generated by Claude-3. The image is sampled from OBELICS dataset.