Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment

TL;DR

The paper tackles the high cost of subjective image quality assessment by introducing uncertainty‑driven sampling for pairwise comparisons. It combines data (aleatoric) and model (epistemic) uncertainties using a deep learning model to estimate the probability of preference and MC‑dropout to quantify prediction reliability, enabling offline preselection of informative pairs via an expected information change criterion. The LBPS framework, especially LBPS‑EIC, demonstrates state‑of‑the‑art performance on PieAPP and PC‑IQA datasets with far fewer human judgments, validated through BT aggregation and multiple accuracy metrics. This approach promises scalable, precise benchmarking and improved training data for learning‑based quality metrics, with potential extensions to reinforcement learning and vision transformers. The results highlight a practical path to efficient, high‑fidelity subjective assessment in image processing domains.

Abstract

Assessing image quality is crucial in image processing tasks such as compression, super-resolution, and denoising. While subjective assessments involving human evaluators provide the most accurate quality scores, they are impractical for large-scale or continuous evaluations due to their high cost and time requirements. Pairwise comparison subjective assessment tests, which rank image pairs instead of assigning scores, offer more reliability and accuracy but require numerous comparisons, leading to high costs. Although objective quality metrics are more efficient, they lack the precision of subjective tests, which are essential for benchmarking and training learning-based quality metrics. This paper proposes an uncertainty-based sampling method to optimize the pairwise comparison subjective assessment process. By utilizing deep learning models to estimate human preferences and identify pairs that need human labeling, the approach reduces the number of required comparisons while maintaining high accuracy. The key contributions include modeling uncertainty for accurate preference predictions and for pairwise sampling. The experimental results demonstrate superior performance of the proposed approach compared to traditional active sampling methods. Software is publicly available at: shimamohammadi/LBPS-EIC

Figures (9)

• Figure 1: Quality difference, $Q(A)-Q(B)$, distributions. The shaded area under the probability density function (PDF) curve of $Q(A)-Q(B)$ represents the probability of preference $\hat{Pr}(A \succ B)$.
• Figure 2: Overview of the pairwise sampling procedure.
• Figure 3: Proposed architecture of the deep learning framework ($f_{\mu}()$ and $f_{\sigma}()$).
• Figure 4: Predicted preference versus ground truth preference using a scatter plot combined with a heatmap. The heatmap overlays the scatter plot to illustrate the density of data points.
• Figure 5: Model uncertainty versus predicted preferences.