Beware of Validation by Eye: Visual Validation of Linear Trends in Scatterplots

TL;DR

This study investigates the effectiveness of visual validation of linear trends in scatterplots and the impact of common visual augmentations. Across two experiments, participants were more accurate at visually estimating slopes than at validating a shown trend, with a bias toward slopes that are too steep and evidence that people implicitly follow orthogonal distance regression (ODR) rather than ordinary least squares (OLS). Adding designs such as error lines, bounding boxes, or confidence intervals reduced some bias but did not improve overall validation accuracy, and in some cases increased task difficulty and response time. The findings call for caution in relying on visual validation for linear trends and highlight the need for better perceptual tools or guidance to support model validation in scatterplots.

Abstract

Visual validation of regression models in scatterplots is a common practice for assessing model quality, yet its efficacy remains unquantified. We conducted two empirical experiments to investigate individuals' ability to visually validate linear regression models (linear trends) and to examine the impact of common visualization designs on validation quality. The first experiment showed that the level of accuracy for visual estimation of slope (i.e., fitting a line to data) is higher than for visual validation of slope (i.e., accepting a shown line). Notably, we found bias toward slopes that are "too steep" in both cases. This lead to novel insights that participants naturally assessed regression with orthogonal distances between the points and the line (i.e., ODR regression) rather than the common vertical distances (OLS regression). In the second experiment, we investigated whether incorporating common designs for regression visualization (error lines, bounding boxes, and confidence intervals) would improve visual validation. Even though error lines reduced validation bias, results failed to show the desired improvements in accuracy for any design. Overall, our findings suggest caution in using visual model validation for linear trends in scatterplots.

Figures (14)

• Figure 1: Example of the same amount of positive and negative slope deviation in the validation task.
• Figure 2: Comparison of validation and estimation accuracy (absolute deviation) with respect to OLS regression. Blue line: Cumulative distribution (CDF) for the estimation errors. Orange line: Logistic regression for the validation acceptance. Green line: Statistical 95% CI. Notice that more statistically valid lines were estimated than accepted by validation and more invalid lines were accepted than estimated.
• Figure 3: Comparison of the validation acceptance rates of positive and negative deviations for positive and negative trends with respect to OLS regression. Green line: statistical 95% CI.
• Figure 4: Histogram of the deviations of the estimated lines for positive and negative trends with respect to OLS regression.
• Figure 5: Illustration of the difference between ordinary least squares (OLS, blue line) and orthogonal distance (ODR, yellow line) regression, adapted from Ciccione.2021.