Modeling Art Evaluations from Comparative Judgments: A Deep Learning Approach to Predicting Aesthetic Preferences

TL;DR

This paper tackles predicting visual-art aesthetic preferences under the challenge of subjective variability and costly labels by adopting comparative learning with pairwise judgments, grounded in Thurstone's Law. It compares a CNN-feature-based deep regression model against a hand-crafted-feature baseline and introduces a dual-branch, hinge-loss pairwise framework that learns from relative preferences without direct ratings. Across computational evaluations and a small human-subject study, deep CNN features substantially outperform the baseline, while comparative learning approaches approach regression performance for representational art and offer substantial gains in annotation efficiency. The results show population-level average ratings are more predictable, personalization remains challenging with limited data, and comparative judgments enable scalable data collection with roughly $60 ext{%}$ faster annotations and $2.5 imes$ more items annotated within the same time budget, making it a practical strategy for large-scale aesthetic modeling.

Abstract

Modeling human aesthetic judgments in visual art presents significant challenges due to individual preference variability and the high cost of obtaining labeled data. To reduce cost of acquiring such labels, we propose to apply a comparative learning framework based on pairwise preference assessments rather than direct ratings. This approach leverages the Law of Comparative Judgment, which posits that relative choices exhibit less cognitive burden and greater cognitive consistency than direct scoring. We extract deep convolutional features from painting images using ResNet-50 and develop both a deep neural network regression model and a dual-branch pairwise comparison model. We explored four research questions: (RQ1) How does the proposed deep neural network regression model with CNN features compare to the baseline linear regression model using hand-crafted features? (RQ2) How does pairwise comparative learning compare to regression-based prediction when lacking access to direct rating values? (RQ3) Can we predict individual rater preferences through within-rater and cross-rater analysis? (RQ4) What is the annotation cost trade-off between direct ratings and comparative judgments in terms of human time and effort? Our results show that the deep regression model substantially outperforms the baseline, achieving up to $328\%$ improvement in $R^2$. The comparative model approaches regression performance despite having no access to direct rating values, validating the practical utility of pairwise comparisons. However, predicting individual preferences remains challenging, with both within-rater and cross-rater performance significantly lower than average rating prediction. Human subject experiments reveal that comparative judgments require $60\%$ less annotation time per item, demonstrating superior annotation efficiency for large-scale preference modeling.

Figures (7)

• Figure 1: Deep neural network regression architecture. Painting images flow from top to bottom: processed through ResNet-50 to extract 2048-dimensional features, fed into encoder $f(\mathbf{x})$ (MLP with BatchNorm and Dropout), and output predicted ratings evaluated with MAE loss.
• Figure 2: Comparative learning framework. Two paintings are processed in parallel from top to bottom: through ResNet-50 for feature extraction, through shared encoder $f(\mathbf{x})$ to produce utility scores $s_i$ and $s_j$, computing difference $C_{ij} = s_i - s_j$, and evaluating against preference label $O_{ij}$ using hinge loss.
• Figure 3: Overall methodology framework. The dataset flows from top to bottom through two feature extraction pipelines (hand-crafted and CNN features), three modeling approaches (Baseline OLS, Enhanced Deep NN, Comparative Pairwise), unified evaluation settings (Average/Within-rater/Cross-rater), and final metrics across four aesthetic tasks.
• Figure 4: Comparative vs. Regression: Abstract Beauty. Pairwise model performance improves with increased sample size $N$ (number of comparison pairs per painting) but remains below regression baseline.
• Figure 5: Comparative vs. Regression: Abstract Liking. Comparative learning shows fluctuating performance below regression baseline.