Scene Perceived Image Perceptual Score (SPIPS): combining global and local perception for image quality assessment

TL;DR

SPIPS addresses the misalignment between traditional pixel‑based IQA and human perception in the era of deep generative editing by disentangling deep features into high‑level semantic and low‑level perceptual streams, and fusing them with traditional IQA maps. The framework uses three modules—a traditional IQA map generator, a deep feature–based perceptual/semantic mapper, and an image quality feature extractor—that feed into a weighted fusion to yield a final quality score via an MLP‑style optimization approach. On the BAPPS dataset, SPIPS achieves superior correlation with human judgments across distortion types, outperforming LPIPS, DISTS, and conventional metrics, with ablation studies confirming the value of semantic information and traditional IQA components. The work demonstrates a practical path toward more human‑aligned IQA by integrating global scene understanding with local perceptual cues, with potential extensions to larger datasets and vision transformers to further model inter‑patch relationships.

Abstract

The rapid advancement of artificial intelligence and widespread use of smartphones have resulted in an exponential growth of image data, both real (camera-captured) and virtual (AI-generated). This surge underscores the critical need for robust image quality assessment (IQA) methods that accurately reflect human visual perception. Traditional IQA techniques primarily rely on spatial features - such as signal-to-noise ratio, local structural distortions, and texture inconsistencies - to identify artifacts. While effective for unprocessed or conventionally altered images, these methods fall short in the context of modern image post-processing powered by deep neural networks (DNNs). The rise of DNN-based models for image generation, enhancement, and restoration has significantly improved visual quality, yet made accurate assessment increasingly complex. To address this, we propose a novel IQA approach that bridges the gap between deep learning methods and human perception. Our model disentangles deep features into high-level semantic information and low-level perceptual details, treating each stream separately. These features are then combined with conventional IQA metrics to provide a more comprehensive evaluation framework. This hybrid design enables the model to assess both global context and intricate image details, better reflecting the human visual process, which first interprets overall structure before attending to fine-grained elements. The final stage employs a multilayer perceptron (MLP) to map the integrated features into a concise quality score. Experimental results demonstrate that our method achieves improved consistency with human perceptual judgments compared to existing IQA models.

Figures (4)

• Figure 1: Qualitative comparison of different metrics against human preference on the BAPPS dataset. SPIPS consistently aligns with human judgment across different distortion types.
• Figure 2: The overall framework of our SPIPS model is structured into three modules: the traditional image quality assessment module (red), the low-level image perception feature assessment module (green), and the high-level image semantic feature assessment module (yellow). SPIPS takes two input images, $I_{eval}$ (the image to be evaluated) and $I_{ref}$ (the reference or ground truth image). The traditional image quality assessment module (red) analyzes quality differences between corresponding regions of $I_{eval}$ and $I_{ref}$ using standard industry metrics such as PSNR, SSIM, and MS-SSIM, producing a quality assessment map. Both $I_{eval}$ and $I_{ref}$ are then fed into a pre-trained deep learning backbone (e.g., AlexNet, VGG, SqueezeNet, represented by the light blue dotted line) to extract feature maps from each layer before the fully connected layers. These feature maps are categorized based on their focus: low-level image perception features (derived from all layers except the last two) and high-level semantic features (from the final two layers of the feature stack). The mean squared error (MSE) between the deep feature representations of $I_{eval}$ and $I_{ref}$ is then computed to generate a deep evaluation map. Each evaluation map undergoes independent quality enhancement before being combined into the final image quality score of $I_{eval}$ through a weighted averaging process. Similar to the optimization approach in the LPIPS model zhang2018unreasonable, this score is compared to human visual perception scores to compute a loss value. The SPIPS model parameters are iteratively optimized via backpropagation, following a similar training strategy to that used in the LPIPS model.
• Figure 3: Qualitative comparison between the full SPIPS model and its ablated variant without the semantic module. SPIPS0 and SPIPS1 denote the evaluation scores assigned by the SPIPS model to $image0$ and $image1$, respectively. The images $image0$ and $image1$ are the two candidates being compared, with $reference$ serving as the ground truth. $ablation0$ and $ablation1$ represent the evaluation scores for $image0$ and $image1$ produced by an ablated version of the SPIPS model, which excludes the semantic module. Human preference: $image0 < image1$
• Figure 4: Qualitative comparison of the full SPIPS model and its ablated variant without traditional IQA metrics such as PSNR and SSIM. The roles of SPIPS0, SPIPS1, $image0$, $reference$, and $image1$ are consistent with those in Figure \ref{['fig:ablation1_comparison']}. Similarly, $ablation0$ and $ablation1$ represent the evaluation scores of $image0$ and $image1$ generated by the ablated version of the SPIPS model. Unlike Figure \ref{['fig:ablation1_comparison']}, however, the ablation variant used here excludes traditional IQA metrics. Human preference: $image0 > image1$