Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Image Quality Assessment Ability of LMMs via Retrieval-Augmented Generation

Kang Fu, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Jun Zhao, Xiongkuo Min, Jia Wang, Guangtao Zhai

TL;DR

This paper addresses improving image quality assessment with Large Multimodal Models without expensive fine-tuning. It introduces IQARAG, a training-free Retrieval-Augmented Generation framework that uses semantically similar reference images annotated with Mean Opinion Scores (MOS) to anchor IQA judgments. IQARAG operates in three phases—Retrieval Feature Extraction, Image Retrieval, and Integration & Quality Score Generation—to produce a final quality score from quality-related tokens in the LMM output. Across four standard IQA datasets and multiple LMMs, IQARAG yields significant IQA performance gains, demonstrating a resource-efficient alternative to fine-tuning for aligning LMM outputs with perceptual quality distributions.

Abstract

Large Multimodal Models (LMMs) have recently shown remarkable promise in low-level visual perception tasks, particularly in Image Quality Assessment (IQA), demonstrating strong zero-shot capability. However, achieving state-of-the-art performance often requires computationally expensive fine-tuning methods, which aim to align the distribution of quality-related token in output with image quality levels. Inspired by recent training-free works for LMM, we introduce IQARAG, a novel, training-free framework that enhances LMMs' IQA ability. IQARAG leverages Retrieval-Augmented Generation (RAG) to retrieve some semantically similar but quality-variant reference images with corresponding Mean Opinion Scores (MOSs) for input image. These retrieved images and input image are integrated into a specific prompt. Retrieved images provide the LMM with a visual perception anchor for IQA task. IQARAG contains three key phases: Retrieval Feature Extraction, Image Retrieval, and Integration & Quality Score Generation. Extensive experiments across multiple diverse IQA datasets, including KADID, KonIQ, LIVE Challenge, and SPAQ, demonstrate that the proposed IQARAG effectively boosts the IQA performance of LMMs, offering a resource-efficient alternative to fine-tuning for quality assessment.

Enhancing Image Quality Assessment Ability of LMMs via Retrieval-Augmented Generation

TL;DR

This paper addresses improving image quality assessment with Large Multimodal Models without expensive fine-tuning. It introduces IQARAG, a training-free Retrieval-Augmented Generation framework that uses semantically similar reference images annotated with Mean Opinion Scores (MOS) to anchor IQA judgments. IQARAG operates in three phases—Retrieval Feature Extraction, Image Retrieval, and Integration & Quality Score Generation—to produce a final quality score from quality-related tokens in the LMM output. Across four standard IQA datasets and multiple LMMs, IQARAG yields significant IQA performance gains, demonstrating a resource-efficient alternative to fine-tuning for aligning LMM outputs with perceptual quality distributions.

Abstract

Large Multimodal Models (LMMs) have recently shown remarkable promise in low-level visual perception tasks, particularly in Image Quality Assessment (IQA), demonstrating strong zero-shot capability. However, achieving state-of-the-art performance often requires computationally expensive fine-tuning methods, which aim to align the distribution of quality-related token in output with image quality levels. Inspired by recent training-free works for LMM, we introduce IQARAG, a novel, training-free framework that enhances LMMs' IQA ability. IQARAG leverages Retrieval-Augmented Generation (RAG) to retrieve some semantically similar but quality-variant reference images with corresponding Mean Opinion Scores (MOSs) for input image. These retrieved images and input image are integrated into a specific prompt. Retrieved images provide the LMM with a visual perception anchor for IQA task. IQARAG contains three key phases: Retrieval Feature Extraction, Image Retrieval, and Integration & Quality Score Generation. Extensive experiments across multiple diverse IQA datasets, including KADID, KonIQ, LIVE Challenge, and SPAQ, demonstrate that the proposed IQARAG effectively boosts the IQA performance of LMMs, offering a resource-efficient alternative to fine-tuning for quality assessment.
Paper Structure (20 sections, 11 equations, 3 figures, 4 tables)

This paper contains 20 sections, 11 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Quality Assessment
  4. Retrieval Augmented Generation
  5. Proposed Method
  6. Retrieval Feature Extraction
  7. Image Retrieval
  8. Integration & Quality Score Generation
  9. Experiment
  10. Datasets
  11. Implementation Details
  12. Evaluation Criteria
  13. Experimental Setup
  14. Prompt setting
  15. Experiment results
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Motivation of IQARAG. By referencing multiple semantically similar but quality-variant images, the people can evaluate image quality more accurate and reliable.
  • Figure 2: Overview of the IQARAG framework. It comprises three phases: (1) Retrieval Feature Extraction, which encodes images into unified visual features; (2) Image Retrieval, which identifies semantically similar but quality-variant references from a curated set; and (3) Integration & Quality Score Generation, which feeds the input, retrieved images, and their MOSs into an LMM to derive the final quality score via quality-related tokens. In phase (3), the dotted/solid line indicates the input prompt without/with IQARAG.
  • Figure 3: Example logits of quality-related tokens (Excellent, Good, Fair, Poor, Bad). The S means quality score. The green, blue and red data indicates without/with IQARAG and ground-truth respectively. It indicates that employing the IQARA framework aligns the logits distribution of quality-related tokens more closely with the ground-truth quality score distribution.