Transforming Video Subjective Testing with Training, Engagement, and Real-Time Feedback

TL;DR

This work tackles the reliability and scalability challenges of subjective video quality assessment by integrating automated participant training, real-time attention monitoring using dynamically evolving golden pairs, and an efficient chain-based pairwise comparison protocol to recover $JOD$ scores with $O(n)$ complexity. The method demonstrates substantial data-quality gains: training increases attentiveness and reduces ties, while real-time feedback yields the most decisive and monotonic quality judgments. A dynamic golden-pair strategy further strengthens attention signals, and the chain-based design dramatically reduces test burden, enabling large-scale deployment in industrial settings. Collectively, the framework supports reliable subjective testing with smaller participant pools and offers practical pathways for deploying robust video quality metrics across codecs and platforms.

Abstract

Subjective video quality assessment is crucial for optimizing streaming and compression, yet traditional protocols face limitations in capturing nuanced perceptual differences and ensuring reliable user input. We propose an integrated framework that enhances rater training, enforces attention through real-time scoring, and streamlines pairwise comparisons to recover quality scores with fewer comparisons. Participants first undergo an automated training quiz to learn key video quality indicators (e.g., compression artifacts) and verify their readiness. During the test, a real-time attention scoring mechanism, using "golden" video pairs, monitors and reinforces rater focus by applying penalties for lapses. An efficient chain-based pairwise comparison procedure is then employed, yielding quality scores in Just-Objectionable-Differences (JOD) units. Experiments comparing three groups (no training, training without feedback, and training with feedback) with 80 participants demonstrate that training-quiz significantly improves data quality in terms of golden unit accuracy and reduces tie rate, while real-time feedback further improves data quality and yields the most monotonic quality ratings. The new training, quiz, testing with feedback, 3-phase approach can significantly reduce the non-monotonic cases on the high quality part of the R-Q curve where normal viewer typically prefer the slightly compressed less-grainy content and help train a better objective video quality metric.

Figures (4)

• Figure 1: Pairwise comparison strategy.
• Figure 2: Dynamically selected golden pairs (green markers) from Groups B and C based on response consensus ($|\bar{r}| > 0.5$, $\sigma_r < 0.3$, and $\geq$75% agreement). Non-golden pairs are shown as red markers.
• Figure 3: Average attention scores (top) and derivative (bottom) across 60 rated pairs. Trained groups (B and C) significantly outperformed Group A ($p < 0.001$). Group C showed significantly lower tie rates than Groups A and B ($p < 0.001$ and $p < 0.01$).
• Figure 4: Recovered JOD scores (top) and PCM distributions (bottom) for a grainy blast scene across Groups A, B, and C.