Learning to Pay Attention: Unsupervised Modeling of Attentive and Inattentive Respondents in Survey Data

TL;DR

A unified, label-free framework for inattentiveness detection that scores response coherence using complementary unsupervised views: geometric reconstruction of Autoencoders and probabilistic dependency modeling (Chow-Liu trees) and probabilistic dependency modeling (Chow-Liu trees).

Abstract

The integrity of behavioral and social-science surveys depends on detecting inattentive respondents who provide random or low-effort answers. Traditional safeguards, such as attention checks, are often costly, reactive, and inconsistent. We propose a unified, label-free framework for inattentiveness detection that scores response coherence using complementary unsupervised views: geometric reconstruction (Autoencoders) and probabilistic dependency modeling (Chow-Liu trees). While we introduce a "Percentile Loss" objective to improve Autoencoder robustness against anomalies, our primary contribution is identifying the structural conditions that enable unsupervised quality control. Across nine heterogeneous real-world datasets, we find that detection effectiveness is driven less by model complexity than by survey structure: instruments with coherent, overlapping item batteries exhibit strong covariance patterns that allow even linear models to reliably separate attentive from inattentive respondents. This reveals a critical ``Psychometric-ML Alignment'': the same design principles that maximize measurement reliability (e.g., internal consistency) also maximize algorithmic detectability. The framework provides survey platforms with a scalable, domain-agnostic diagnostic tool that links data quality directly to instrument design, enabling auditing without additional respondent burden.

Figures (7)

• Figure 1: A toy example illustrating the proposed approach. Survey responses from multiple participants are represented as structured categorical data across questions (e.g., $\langle \textit{Age}, \textit{Height}, \textit{Sex}, \textit{Weight} \rangle$). The model first learns to reconstruct each participant’s responses (Task 1) and, based on reconstruction errors, detects incoherent (hard-to-model) responses. We hypothesize that participants with many such inconsistencies are inattentive (Task 2). Green denotes attentive respondents or well-reconstructed answers; red indicates inattentive respondents or poorly reconstructed patterns. Together, the two tasks demonstrate how unsupervised reconstruction enables automated detection of low-quality survey data.
• Figure 2: Architecture of a simple Autoencoder. The autoencoder consists of two parts: (a) the Encoder, which encodes the information into latent variables, and (b) the Decoder, which decodes the information to the initial input. https://www.digitalocean.com/community/tutorials/autoencoder-image-compression-keras
• Figure 3: Percentile Loss trade-off: $\Delta \text{Lift}$ (reconstruction) vs. $p$ relative to $p{=}100$. Medians are negative at $p{=}80/85/90$ and approach $0$ at $p{=}95$, indicating reconstruction improves monotonically as $p\!\to\!100$.
• Figure 4: Percentile Loss trade-off: $\Delta \text{AUC}$ (randomness detection) vs. $p$ relative to $p{=}100$. Gains at $p{=}80/85/90$ and attenuation at $p{=}95$ suggest a broad optimum near $p\approx 85\text{--}90$.
• Figure 5: Toy Chow--Liu tree: height as root; edges reflect strongest mutual information.