HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing

TL;DR

HMGIE introduces a hierarchical, multi-grained framework for visual-textual inconsistency evaluation to cleanse vision-language data. It integrates a semantic graph generation module, a hierarchical inconsistency evaluation graph (HIEG) module with progressive QA-based reasoning, and a quantitative evaluation module that yields H-Scores for semantic accuracy and completeness. The approach is validated on the MVTID dataset, a four-granularity benchmark, and extended benchmarks, demonstrating superior detection performance and better interpretability than existing methods. The work advances data quality for vision-language models and multimodal systems by enabling robust, granular inconsistency detection and explanation.

Abstract

Visual-textual inconsistency (VTI) evaluation plays a crucial role in cleansing vision-language data. Its main challenges stem from the high variety of image captioning datasets, where differences in content can create a range of inconsistencies (\eg, inconsistencies in scene, entities, entity attributes, entity numbers, entity interactions). Moreover, variations in caption length can introduce inconsistencies at different levels of granularity as well. To tackle these challenges, we design an adaptive evaluation framework, called Hierarchical and Multi-Grained Inconsistency Evaluation (HMGIE), which can provide multi-grained evaluations covering both accuracy and completeness for various image-caption pairs. Specifically, the HMGIE framework is implemented by three consecutive modules. Firstly, the semantic graph generation module converts the image caption to a semantic graph for building a structural representation of all involved semantic items. Then, the hierarchical inconsistency evaluation module provides a progressive evaluation procedure with a dynamic question-answer generation and evaluation strategy guided by the semantic graph, producing a hierarchical inconsistency evaluation graph (HIEG). Finally, the quantitative evaluation module calculates the accuracy and completeness scores based on the HIEG, followed by a natural language explanation about the detection results. Moreover, to verify the efficacy and flexibility of the proposed framework on handling different image captioning datasets, we construct MVTID, an image-caption dataset with diverse types and granularities of inconsistencies. Extensive experiments on MVTID and other benchmark datasets demonstrate the superior performance of the proposed HMGIE to current state-of-the-art methods.

Figures (13)

• Figure 1: Comparison of inconsistency evaluation methods on captions with various granularities. Captions range from short to detailed, containing different types of inconsistencies (object error in $T_2$, scene error in $T_3$, and attribute error in $T_4$). Existing methods show limitations: CLIPScore fails to differentiate between consistent and inconsistent short captions ($T_1$vs.$T_2$), while VDC misses inconsistencies in longer captions ($T_3$, $T_4$). In contrast, HMGIE provides more reasonable h-scores: $\mathcal{H}_{acc}$ for semantic accuracy and $\mathcal{H}_{comp}$ for semantic completeness.
• Figure 2: The overall illustration of our proposed hierarchical and multi-grained inconsistency evaluation (HMGIE) framework.
• Figure 3: Ablation study on semantic graph (SG) in HMGIE: (a) TPR and (b) FPR comparison across granularity levels.
• Figure 4: Impact of different LLMs and MLLMs.
• Figure 5: ROUGE-4 for captions before and after repair.