VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models

TL;DR

This work tackles dirty-sample detection in data-centric AI by introducing Versatile Data Cleanser (VDC), a universal detector that leverages multimodal large language models to measure visual-linguistic inconsistency between images and labels. The method uses a three-stage pipeline—Visual Question Generation, Visual Question Answering, and Visual Answer Evaluation—to quantify semantic alignment via a vote-based threshold, enabling training-free detection across poisoned samples, noisy labels, and their hybrids. Across CIFAR-10, ImageNet-100, and ImageNet-Dog, VDC consistently outperforms specialized detectors and improves downstream robustness when purified data are used for retraining. The approach demonstrates strong generalization to multiple dirty-sample types and offers a practical, scalable solution for data cleaning in real-world AI systems, with performance that is expected to improve as LLMs continue to mature.

Abstract

The role of data in building AI systems has recently been emphasized by the emerging concept of data-centric AI. Unfortunately, in the real-world, datasets may contain dirty samples, such as poisoned samples from backdoor attack, noisy labels in crowdsourcing, and even hybrids of them. The presence of such dirty samples makes the DNNs vunerable and unreliable.Hence, it is critical to detect dirty samples to improve the quality and realiability of dataset. Existing detectors only focus on detecting poisoned samples or noisy labels, that are often prone to weak generalization when dealing with dirty samples from other domains.In this paper, we find a commonality of various dirty samples is visual-linguistic inconsistency between images and associated labels. To capture the semantic inconsistency between modalities, we propose versatile data cleanser (VDC) leveraging the surpassing capabilities of multimodal large language models (MLLM) in cross-modal alignment and reasoning.It consists of three consecutive modules: the visual question generation module to generate insightful questions about the image; the visual question answering module to acquire the semantics of the visual content by answering the questions with MLLM; followed by the visual answer evaluation module to evaluate the inconsistency.Extensive experiments demonstrate its superior performance and generalization to various categories and types of dirty samples. The code is available at \url{https://github.com/zihao-ai/vdc}.

Figures (12)

• Figure 1: The framework of Versatile Data Cleanser. Given the image and label, the visual question generation module first generates general and label-specific questions respectively. Then the visual question answering module answers the generated questions based on the image. Last, the visual question evaluation module evaluates the correctness of answers and makes the final judge based on the vote-based ensemble.
• Figure 2: Effect of question types.
• Figure 3: Effect of question numbers.
• Figure 4: Effect of MLLM.
• Figure 6: The Hello Kitty pattern used in Blended.