Simulation-based Optimization for Augmented Reading

TL;DR

This work proposes framing augmented reading as a simulation-based optimization problem grounded in resource-rational models of human reading, and introduces two complementary optimization pipelines: an offline approach that explores design alternatives using simulated readers, and an online approach that personalizes reading interfaces in real time using ongoing interaction data.

Abstract

Augmented reading systems aim to adapt text presentation to improve comprehension and task performance, yet existing approaches rely heavily on heuristics, opaque data-driven models, or repeated human involvement in the design loop. We propose framing augmented reading as a simulation-based optimization problem grounded in resource-rational models of human reading. These models instantiate a simulated reader that allocates limited cognitive resources, such as attention, memory, and time under task demands, enabling systematic evaluation of text user interfaces. We introduce two complementary optimization pipelines: an offline approach that explores design alternatives using simulated readers, and an online approach that personalizes reading interfaces in real time using ongoing interaction data. Together, this perspective enables adaptive, explainable, and scalable augmented reading design without relying solely on human testing.

Figures (3)

• Figure 1: Resource-rational reading models enable principled optimization of text user interfaces across a wide design space. (Left) A reading agent interacts with text-based user interfaces by allocating visual attention over time, guided by perceptual input, memory state, task objectives, and an adaptive policy. (Right) A design space for augmented reading defined by user groups, text presentation designs, and task demands. The same underlying model can be instantiated to evaluate diverse reading scenarios, illustrated here with three examples: (top) a reader with restricted visual field (tunnel vision), (middle) reading supported by visually salient text cues (e.g., font size, type, and color), and (bottom) multitasking contexts such as reading while driving from an in-vehicle display. Together, the model enables principled comparison and optimization of text UI designs across users, tasks, and environments.
• Figure 2: Optimization paradigms for augmented reading. (a) Human-in-the-loop optimization. Traditional text UI design relies on human readers to generate behavioral data and feedback, making optimization costly and difficult to scale. (b) Simulation-based offline optimization. A simulated reading agent represents human readers to evaluate and optimize candidate text UIs, enabling systematic exploration of the design space without repeated user studies. (c) Simulation-based online optimization. Human interaction data are used to initialize a personalized simulated agent that continues reading on the user’s behalf, supporting real-time evaluation and adaptive UI optimization during ongoing reading.
• Figure 3: Gallery of simulation-based optimization for augmented reading. Three scenarios illustrate how a simulated reading agent predicts human-like reading behavior (red dots indicate fixations on text; blue dots indicate attention to the surrounding environment) and task performance (summarized by model-predicted metrics in orange boxes) under different contexts, and how these predictions guide UI optimization. Across (a) reading while walking on AR, (b) time-limited reading, and (c) visual search while driving, the model identifies bottlenecks in workload, comprehension, or safety. Targeted design interventions, such as increasing layout sparsity, reducing content load, or enhancing visual saliency lead to improved predicted reading behavior and task performance after optimization. (d) Individualized reading support demonstrates how reader-specific simulation reveals inefficient lexical access (e.g., slow reading speed and dense fixations), enabling personalized adaptations such as enhanced lexical saliency to improve predicted reading speed and comprehension.