Déjà Vu? Decoding Repeated Reading from Eye Movements

TL;DR

This paper tackles decoding whether a text has been previously read from eye movements during reading. It introduces two prediction tasks (single-trial and paired-trials) and deploys both feature-based (notably XGBoost) and neural multimodal models, augmented with synthetic scanpaths generated by the cognitive model E-Z Reader. Across the OneStop Eye Movements dataset, feature-based approaches typically outperform neural models, achieving up to ~70% accuracy in single-trial and ~91% in paired-trials, with synthetic scanpath augmentation offering occasional gains. The work advances understanding of memory effects in reading as captured by eye movements and discusses practical uses in education and content personalization, while addressing ethical and privacy concerns for deploying such predictive systems.

Abstract

Be it your favorite novel, a newswire article, a cooking recipe or an academic paper -- in many daily situations we read the same text more than once. In this work, we ask whether it is possible to automatically determine whether the reader has previously encountered a text based on their eye movement patterns. We introduce two variants of this task and address them with considerable success using both feature-based and neural models. We further introduce a general strategy for enhancing these models with machine generated simulations of eye movements from a cognitive model. Finally, we present an analysis of model performance which on the one hand yields insights on the information used by the models, and on the other hand leverages predictive modeling as an analytic tool for better characterization of the role of memory in repeated reading. Our work advances the understanding of the extent and manner in which eye movements in reading capture memory effects from prior text exposure, and paves the way for future applications that involve predictive modeling of repeated reading.

Figures (6)

• Figure 1: Examples of eye movements over a single passage; top: first reading, bottom: repeated reading. Circles represent fixations, and lines represent saccades.
• Figure 2: First and repeated reading for one participant. After reading a 10-article batch in a random order of articles, there is a consecutive repeated reading of the last article in position 10, and then a non-consecutive repeated reading of one of the articles in positions 1-9.
• Figure 3: Visualization of a 10-article, 60-participant data split, divided into train, validation, and three test regimes. Each non-empty cell represents a participant-article pair, comprising the first and repeated readings of an article by the same participant. '*X' denotes consecutive repeated reading and '*X' denotes non-consecutive repeated reading (i.e. with intervening articles between the first and second readings).
• Figure 4: Analysis of the E-Z Reader augmented XGBoost model's behavior as a function of item position. Depicted are probability assignment (top) and classification accuracy (bottom) with 95% confidence intervals. (a) First and repeated reading (RR) as a function of article position in the experiment. (b) Repeated reading as a function of the article position in the first reading. See \ref{['fig:exp-schema']} for the experiment structure.
• Figure 5: Visualization of two Saccade Networks as defined in \ref{['sec:feature_based_methods']}.The top network represents the first reading, while the bottom network corresponds to the repeated reading of the same paragraph by the same participant. Different colors indicate different sentences within the paragraph.