Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization

TL;DR

This work investigates whether deep-feature-based perceptual similarity metrics, trained on natural images, can meaningfully approximate human judgments of information visualizations. By conceptually replicating two crowd-sourced studies (scatterplots and visual encodings) and comparing against traditional metrics, the authors show that ImageNet-pretrained weights often match or exceed gradient-tuned MS-SSIM in aligning with human judgments, highlighting successful cross-domain transfer. However, results also reveal limitations: deep features struggle with color and glyph-shape similarities, and effects of Stylized ImageNet weights are not consistently beneficial. The findings suggest practical potential for transfer-learned perceptual metrics to narrow visualization design spaces and inform automated evaluation, while emphasizing careful consideration of task type and visual primitives when applying these metrics.

Abstract

Judging the similarity of visualizations is crucial to various applications, such as visualization-based search and visualization recommendation systems. Recent studies show deep-feature-based similarity metrics correlate well with perceptual judgments of image similarity and serve as effective loss functions for tasks like image super-resolution and style transfer. We explore the application of such metrics to judgments of visualization similarity. We extend a similarity metric using five ML architectures and three pre-trained weight sets. We replicate results from previous crowd-sourced studies on scatterplot and visual channel similarity perception. Notably, our metric using pre-trained ImageNet weights outperformed gradient-descent tuned MS-SSIM, a multi-scale similarity metric based on luminance, contrast, and structure. Our work contributes to understanding how deep-feature-based metrics can enhance similarity assessments in visualization, potentially improving visual analysis tools and techniques. Supplementary materials are available at https://osf.io/dj2ms.

Figures (20)

• Figure 1: A general framework for comparing two information-processing systems (e.g., humans vs. machines), inspired by the framework proposed by Sucholutsky et al. sucholutsky2023getting. Framework uses Demiralp et al.'s experiment on the perception of shape similarity as one concrete example demiralp2014learning.
• Figure 2: Diagram showing how the deep-feature-based perceptual similarity metric calculates "perceptual distance", using VGG16 as the DL network $\mathcal{F}$.
• Figure 3: Screenshot obtained from the supplementary materials of Pandey et al. pandey2016towards showing Participant 1's final screen after completing the naming phase, with group descriptions and their corresponding easiness and confidence scores (separated by pipe characters).
• Figure 4: Black diamonds represent Veras and Collins' veras2019discriminability results using gradient-descent-tuned MS-SSIM. Crosses represent results using Mean Squared Error (MSE). Points are offset for clarity.
• Figure 5: Black diamonds represent Veras and Collins' veras2019discriminability results using gradient-descent-tuned MS-SSIM. Crosses represent results using MSE. Dots show means of 10 trials. Lines indicate 95% confidence intervals (CIs) from non-parametric bootstrap. Points and CIs are slightly offset for clarity.