Q-Adapt: Adapting LMM for Visual Quality Assessment with Progressive Instruction Tuning

TL;DR

The paper tackles Explainable Image Quality Assessment by addressing conflicts between two explanation tasks using Large Multi-modal Foundation Models. It introduces Q-Adapt, a progressive instruction-tuning framework consisting of a universal perception knowledge learning stage with LoRA and an instruction-guided visual prompting stage employing a bi-directional V-T Generator and T-V Prompter to dynamically adapt visual features to task instructions. This approach yields a lightweight 3B-parameter model with LoRA that achieves competitive or superior results on perceptual benchmarks and widely used IQA datasets, while reducing task conflicts between overall quality explanations and attribute-wise perception answering. The work demonstrates that structured, stage-wise adaptation with flexible visual prompts enhances explainable IQA and offers practical tooling for efficient EIQA deployment in real-world applications.

Abstract

The rapid advancement of Large Multi-modal Foundation Models (LMM) has paved the way for the possible Explainable Image Quality Assessment (EIQA) with instruction tuning from two perspectives: overall quality explanation, and attribute-wise perception answering. However, existing works usually overlooked the conflicts between these two types of perception explanations during joint instruction tuning, leading to insufficient perception understanding. To mitigate this, we propose a new paradigm for perception-oriented instruction tuning, i.e., Q-Adapt, which aims to eliminate the conflicts and achieve the synergy between these two EIQA tasks when adapting LMM, resulting in enhanced multi-faceted explanations of IQA. Particularly, we propose a progressive instruction tuning strategy by dividing the adaption process of LMM for EIQA into two stages, where the first stage empowers the LMM with universal perception knowledge tailored for two tasks using an efficient transfer learning strategy, i.e., LoRA, and the second stage introduces the instruction-adaptive visual prompt tuning to dynamically adapt visual features for the different instructions from two tasks. In this way, our proposed Q-Adapt can achieve a lightweight visual quality evaluator, demonstrating comparable performance and, in some instances, superior results across perceptual-related benchmarks and commonly-used IQA databases. The source code is publicly available at https://github.com/yeppp27/Q-Adapt.

Figures (6)

• Figure 1: The effect of different task instruction tuning for quality explanation task.
• Figure 2: The effect of different task instruction tuning for perception answering task.
• Figure 3: The overview for our proposed Q-Adapt, which employs progressive instruction tuning to achieve the synergy between two EIQA tasks. Concretely, the progressive instruction tuning strategy comprises two stages: the universal perception knowledge requiring stage (i.e., the first stage) tailored for building a powerful base for two tasks, and the instruction-adaptive visual prompting stage for dynamically adapting visual features for task instruction. Additionally, the second stage incorporates the V-T Generator and T-V Prompter to achieve the bi-directional multimodal interactions.
• Figure 4: The visualizations of the original visual feature and instruction-adaptive visual prompt. (a)-(d) illustrate results from Q-bench, while (e)-(h) show results from Q-bench2. And "Task1" refers to the attribute-wise perception answering task, "Task2" denotes the overall quality explanation task.
• Figure 5: The effect of task selection in progressive instruction tuning for explanation task.