Image Quality Assessment: Unifying Structure and Texture Similarity

TL;DR

This work introduces DISTS, a full-reference image quality index that tolerates texture resampling by coupling a texture-focused statistical representation with a structure-based similarity. It builds an injective, differentiable perceptual transform from a modified VGG backbone, uses global channel-mean statistics to model texture and a CS-like covariance term for structure, and learns weights to align with human judgments while enforcing texture invariance. Empirical results show strong performance on standard IQA datasets, robust texture similarity and retrieval, and notable invariance to mild geometric changes, all without requiring texture-specific training data. The approach is positioned as a versatile tool for perceptual optimization in image processing pipelines and texture-focused analyses.

Abstract

Objective measures of image quality generally operate by comparing pixels of a "degraded" image to those of the original. Relative to human observers, these measures are overly sensitive to resampling of texture regions (e.g., replacing one patch of grass with another). Here, we develop the first full-reference image quality model with explicit tolerance to texture resampling. Using a convolutional neural network, we construct an injective and differentiable function that transforms images to multi-scale overcomplete representations. We demonstrate empirically that the spatial averages of the feature maps in this representation capture texture appearance, in that they provide a set of sufficient statistical constraints to synthesize a wide variety of texture patterns. We then describe an image quality method that combines correlations of these spatial averages ("texture similarity") with correlations of the feature maps ("structure similarity"). The parameters of the proposed measure are jointly optimized to match human ratings of image quality, while minimizing the reported distances between subimages cropped from the same texture images. Experiments show that the optimized method explains human perceptual scores, both on conventional image quality databases, as well as on texture databases. The measure also offers competitive performance on related tasks such as texture classification and retrieval. Finally, we show that our method is relatively insensitive to geometric transformations (e.g., translation and dilation), without use of any specialized training or data augmentation. Code is available at https://github.com/dingkeyan93/DISTS.

Figures (11)

• Figure 1: Existing full-reference IQA models are overly sensitive to point-by-point deviations between images of the same texture. (a) A grass image and (b) the same image, distorted by JPEG compression. (c) Resampling of the same grass as in (a). Popular IQA measures, including PSNR, SSIM wang2004image, FSIM zhang2011fsim, VIF sheikh2006image, GMSD xue2014gradient, DeepIQA bosse2018deep, PieAPP prashnani2018pieapp, and LPIPS zhang2018unreasonable, predict that image (b) has a better perceived quality than image (c), which is in disagreement with human rating. In contrast, the proposed DISTS model makes the correct prediction. (Zoom in to improve visibility of details).
• Figure 2: Recovery of a reference image by optimization of IQA measures. Recovery is implemented by solving $y^\star =\mathop{\mathrm{arg\,min}}\limits_y D(x,y)$ with gradient descent, where $D$ is an IQA distortion measure and $x$ is a given reference image. (a) Reference image. (b) Corrupted initial image $y_0$, obtained by compressing the reference image using JPEG at a low bitrate. (c)-(f) Images recovered from (b) by optimizing different metrics (as indicated). (g) Corrupted initial image, obtained by adding white Gaussian noise. (h)-(k) Images recovered from (g) by optimizing indicated metrics. In all cases, the optimization converges, yielding a distortion score substantially lower than that of the initial.
• Figure 3: Images synthesized to match the mean values of channels up to a given layer (top) or from individual layers (bottom) of the pre-trained VGG network. (a) Reference texture. (b) Up to $\text{conv1}\_\text{2}$. (c) Up to $\text{conv2}\_\text{2}$. (d) Up to $\text{conv3}\_\text{3}$. (e) Up to $\text{conv4}\_\text{3}$. (f) Up to $\text{conv5}\_\text{3}$. (g) Only $\text{conv1}\_\text{2}$. (h) Only $\text{conv2}\_\text{2}$. (i) Only $\text{conv3}\_\text{3}$. (j) Only $\text{conv4}\_\text{3}$. (k) Only $\text{conv5}\_\text{3}$.
• Figure 4: Synthesis results for three example texture photographs. (a) Reference textures. (b) Images synthesized using the method of Portilla & Simoncelli portilla2000parametric. (c) Images synthesized using Gatys et al. gatys2015texture. (d) Images synthesized using our texture model (Eq. (\ref{['eq:texture_syn']})).
• Figure 5: Selected feature maps from the six layers of the VGG decomposition of the "buildings" image. (a) Zeroth stage (original image). (b) First stage. (c) Second stage. (d) Third stage. (e) Fourth stage. (f) Fifth stage. The feature map intensities are re-scaled for better visibility.

Theorems & Definitions (2)

• Lemma 1
• proof