Automatic Scoring of Cognition Drawings: Assessing the Quality of Machine-Based Scores Against a Gold Standard

TL;DR

This study investigates automated scoring of cognitive-drawing tests from population surveys to improve data quality and reduce interviewer burden. Using CNNs, notably ConvNeXt Base, it compares training on interviewer scores versus a gold-standard labeling procedure, finding about a $9.5$ percentage point improvement when trained on gold-standard data, with an overall accuracy near $85\%$ that surpasses human interviewers by roughly $5$ percentage points. The work demonstrates feasibility and outlines production considerations, limitations, and concrete next steps, including data expansion, active learning, and expert-in-the-loop validation to reach production-grade reliability. The findings have practical implications for large-scale cognitive assessment in surveys, enabling more standardized, scalable, and potentially more accurate dementia screening indicators linked to ACE-III drawings.

Abstract

Figure drawing is often used as part of dementia screening protocols. The Survey of Health Aging and Retirement in Europe (SHARE) has adopted three drawing tests from Addenbrooke's Cognitive Examination III as part of its questionnaire module on cognition. While the drawings are usually scored by trained clinicians, SHARE uses the face-to-face interviewers who conduct the interviews to score the drawings during fieldwork. This may pose a risk to data quality, as interviewers may be less consistent in their scoring and more likely to make errors due to their lack of clinical training. This paper therefore reports a first proof of concept and evaluates the feasibility of automating scoring using deep learning. We train several different convolutional neural network (CNN) models using about 2,000 drawings from the 8th wave of the SHARE panel in Germany and the corresponding interviewer scores, as well as self-developed 'gold standard' scores. The results suggest that this approach is indeed feasible. Compared to training on interviewer scores, models trained on the gold standard data improve prediction accuracy by about 10 percentage points. The best performing model, ConvNeXt Base, achieves an accuracy of about 85%, which is 5 percentage points higher than the accuracy of the interviewers. While this is a promising result, the models still struggle to score partially correct drawings, which are also problematic for interviewers. This suggests that more and better training data is needed to achieve production-level prediction accuracy. We therefore discuss possible next steps to improve the quality and quantity of training examples.

Figures (7)

• Figure 1: Examples of the three drawings: two overlapping infinity loops, a three-dimensional cube, and a clock with its hands set to a specific time.
• Figure 2: Confusion matrix of interviewer scores against gold standard scores. The percentages in the rows show the proportion of interviewer scores in each class, with correct scores highlighted. Absolute numbers are given in parentheses.
• Figure 3: Performance of different model architectures on interviewer scored labels (semi-transparent) and gold standard scored labels (solid). Mean accuracy values across all experiments.
• Figure 4: Effect of hyperparameter variation on accuracy across experiments. Dots represent point estimates with error bars showing 95 percent confidence intervals. Reference categories are: interviewer-scored data, no augmentation, and 50 epochs of training.
• Figure 5: Confusion matrix of the best ConvNeXt Base model trained with data augmentation for 100 epochs on the gold standard data. Percentages in rows show the proportion of model predictions in each class with correct predictions highlighted. Parantheses contain absolute numbers.