Instruction-augmented Multimodal Alignment for Image-Text and Element Matching

TL;DR

The paper tackles the challenge of accurately assessing semantic alignment between text descriptions and images produced by text-to-image models. It introduces iMatch, an instruction-augmented multimodal alignment framework that fine-tunes multimodal large language models on EvalMuse-40K and employs four image-text augmentation strategies (QAlign, validation-set augmentation, element augmentation, image augmentation) plus two element-matching augmentations (prompt type augmentation, score perturbation). The method yields state-of-the-art results on EvalMuse-40K and wins the NTIRE 2025 Track 1 Image-Text Alignment, demonstrating strong performance in both global matching and fine-grained element-level judgments. The approach offers practical benefits for evaluating image generation quality with robust generalization and could inform deployment and benchmarking pipelines in real-world settings.

Abstract

With the rapid advancement of text-to-image (T2I) generation models, assessing the semantic alignment between generated images and text descriptions has become a significant research challenge. Current methods, including those based on Visual Question Answering (VQA), still struggle with fine-grained assessments and precise quantification of image-text alignment. This paper presents an improved evaluation method named Instruction-augmented Multimodal Alignment for Image-Text and Element Matching (iMatch), which evaluates image-text semantic alignment by fine-tuning multimodal large language models. We introduce four innovative augmentation strategies: First, the QAlign strategy creates a precise probabilistic mapping to convert discrete scores from multimodal large language models into continuous matching scores. Second, a validation set augmentation strategy uses pseudo-labels from model predictions to expand training data, boosting the model's generalization performance. Third, an element augmentation strategy integrates element category labels to refine the model's understanding of image-text matching. Fourth, an image augmentation strategy employs techniques like random lighting to increase the model's robustness. Additionally, we propose prompt type augmentation and score perturbation strategies to further enhance the accuracy of element assessments. Our experimental results show that the iMatch method significantly surpasses existing methods, confirming its effectiveness and practical value. Furthermore, our iMatch won first place in the CVPR NTIRE 2025 Text to Image Generation Model Quality Assessment - Track 1 Image-Text Alignment.

Figures (6)

• Figure 1: The instruction set augmentation process of the proposed iMatch.
• Figure 2: The construction process of the element instruction set augmentation. We map elements to integers between 1 and 7 and incorporate them into the user query. Additionally, confidence scores are also integrated into the user query.
• Figure 3: The inference process of the image-text matching augmented model. During inference, we extract closed-set probabilities for rating levels and perform a weighted average to obtain the MLLM-predicted score.
• Figure 4: The process of validation set augmentation. We use the model trained on the training set to generate pseudo-labels for the validation set during the development phase, and then use these pseudo-labeled validation data to augment the training dataset.
• Figure 5: Examples of image augmentation. We employ three types of augmentations: Random Lighting Augmentation, Random Grid Distortion, and Random Crop Augmentation.