Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Consistency Guided Multi-modal Prompt Learning for Blind AI Generated Image Quality Assessment

Jun Fu, Wei Zhou, Qiuping Jiang, Hantao Liu, Guangtao Zhai

TL;DR

This letter introduces learnable textual and visual prompts in language and vision branches of CLIP models, respectively, and designs a text-to-image alignment quality prediction task, whose learned vision-language consistency knowledge is used to guide the optimization of the above multi-modal prompts.

Abstract

Recently, textual prompt tuning has shown inspirational performance in adapting Contrastive Language-Image Pre-training (CLIP) models to natural image quality assessment. However, such uni-modal prompt learning method only tunes the language branch of CLIP models. This is not enough for adapting CLIP models to AI generated image quality assessment (AGIQA) since AGIs visually differ from natural images. In addition, the consistency between AGIs and user input text prompts, which correlates with the perceptual quality of AGIs, is not investigated to guide AGIQA. In this letter, we propose vision-language consistency guided multi-modal prompt learning for blind AGIQA, dubbed CLIP-AGIQA. Specifically, we introduce learnable textual and visual prompts in language and vision branches of CLIP models, respectively. Moreover, we design a text-to-image alignment quality prediction task, whose learned vision-language consistency knowledge is used to guide the optimization of the above multi-modal prompts. Experimental results on two public AGIQA datasets demonstrate that the proposed method outperforms state-of-the-art quality assessment models. The source code is available at https://github.com/JunFu1995/CLIP-AGIQA.

Vision-Language Consistency Guided Multi-modal Prompt Learning for Blind AI Generated Image Quality Assessment

TL;DR

This letter introduces learnable textual and visual prompts in language and vision branches of CLIP models, respectively, and designs a text-to-image alignment quality prediction task, whose learned vision-language consistency knowledge is used to guide the optimization of the above multi-modal prompts.

Abstract

Recently, textual prompt tuning has shown inspirational performance in adapting Contrastive Language-Image Pre-training (CLIP) models to natural image quality assessment. However, such uni-modal prompt learning method only tunes the language branch of CLIP models. This is not enough for adapting CLIP models to AI generated image quality assessment (AGIQA) since AGIs visually differ from natural images. In addition, the consistency between AGIs and user input text prompts, which correlates with the perceptual quality of AGIs, is not investigated to guide AGIQA. In this letter, we propose vision-language consistency guided multi-modal prompt learning for blind AGIQA, dubbed CLIP-AGIQA. Specifically, we introduce learnable textual and visual prompts in language and vision branches of CLIP models, respectively. Moreover, we design a text-to-image alignment quality prediction task, whose learned vision-language consistency knowledge is used to guide the optimization of the above multi-modal prompts. Experimental results on two public AGIQA datasets demonstrate that the proposed method outperforms state-of-the-art quality assessment models. The source code is available at https://github.com/JunFu1995/CLIP-AGIQA.

Paper Structure

This paper contains 12 sections, 12 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Recap of CLIP Models
  4. Text-to-Image Alignment Quality Prediction
  5. Perceptual Quality Prediction
  6. Loss Function
  7. Experiments
  8. Database and Evaluation Criteria
  9. Implement Details
  10. Performance Comparisons
  11. Ablation Study
  12. Conclusion

Figures (2)

  • Figure 1: (a) Comparison between natural images and AI generated images on the AGIQA-1K dataset AGIQA1k; (b) Spearman Rank Correlation Coefficient (SRCC) of the text-to-image alignment quality and perceptual quality on the AIGCIQA-2023 dataset aigciqa2023.
  • Figure 2: Framework of our proposed method. It includes an auxiliary task and a main task. Both tasks are based on CLIP models and involve multi-modal prompt learning. Moreover, learnable visual prompts of the main task are conditioned on those of the auxiliary task.