Decoding Open-Ended Information Seeking Goals from Eye Movements in Reading

TL;DR

This work asks, for the first time, whether open-ended reading goals can be automatically decoded solely from eye movements in reading, and develops and compares several discriminative and generative multimodal text and eye movements LLMs for these tasks.

Abstract

When reading, we often have specific information that interests us in a text. For example, you might be reading this paper because you are curious about LLMs for eye movements in reading, the experimental design, or perhaps you wonder ``This sounds like science fiction. Does it actually work?''. More broadly, in daily life, people approach texts with any number of text-specific goals that guide their reading behavior. In this work, we ask, for the first time, whether open-ended reading goals can be automatically decoded solely from eye movements in reading. To address this question, we introduce goal decoding tasks and evaluation frameworks using large-scale eye tracking for reading data in English with hundreds of text-specific information seeking tasks. We develop and compare several discriminative and generative multimodal text and eye movements LLMs for these tasks. Our experiments show considerable success on the task of selecting the correct goal among several options, and even progress towards free-form textual reconstruction of the precise goal formulation. These results open the door for further scientific investigation of goal driven reading, as well as the development of educational and assistive technologies that will rely on real-time decoding of reader goals from their eye movements.

Figures (7)

• Figure 1: Example of a OneStop paragraph, its three questions, and each question's critical span: the paragraph segment essential for answering the question. See details on notation in \ref{['sec:problem-setting']}.
• Figure 2: Two model types for decoding the question presented to a participant before reading the text, from their eye movements over the text. (a) Discriminative models that score candidate questions and are evaluated on question selection accuracy. (b) Generative models that reconstruct the question presented to the reader, and are evaluated on question reconstruction quality.
• Figure 3: Reading-Time Informed Embedding Similarity models for the question selection task.
• Figure 4: Coefficients from a mixed-effects model which predicts the probability assigned by RoBERTEye-Fixations to the correct question from 11 features of the experimental trial. Predictors are z-normalized, to make the coefficient magnitudes comparable. Coefficient statistical significance after a 11x Bonferroni correction is marked with '*' $p < 0.05$, '**' $p < 0.01$, and '***' $p < 0.001$.
• Figure 5: Goal reconstruction from eye movements evaluations of the finetuned models DalEye-Llama and DalEye-GPT, as well as zero-shot and few-shot Gemini for (1) the identity of the generated question word, (2) UIUC semantic category of the question, (3) BLEU score, (4) BERTScore, (5) downstream QA accuracy based on the answer selection of a multiple-choice QA model. The models are benchmarked against five baselines that do not include eye movements. Two baselines are human-composed questions for a different and the same critical span. Three additional baselines are LLM-generated questions: an arbitrary question about the text generated with Gemini 3, and questions from text-only variants of DalEye-Llama and DalEye-GPT finetuned on the question decoding task using only the text. Results are aggregated over 10 data splits. Presented are intercept coefficients with 95% confidence intervals from linear mixed effects models with random intercepts for participants and paragraphs: $measure \sim 1 + (1 \mid participant) + (1 \mid paragraph)$ for eye movement models and $measure \sim 1 + (1 \mid paragraph)$ for the baselines. The highest score in each combination of evaluation measure and evaluation regime is marked with $\star$.