GazeSummary: Exploring Gaze as an Implicit Prompt for Personalization in Text-based LLM Tasks

TL;DR

The paper investigates using user gaze as an implicit prompt to personalize text-based LLM tasks, addressing the burden of explicit prompts in wearable AI. It evaluates three gaze representations—Gaze Density, Gaze Heatmap, and Attention-based Gaze (SVM)—and identifies gaze heatmaps as the most effective for fine-grained, phrase-level personalization. A prototype gaze-guided academic reading assistant demonstrates automatic personalized summaries, with a user study (n=10) showing gaze-based summaries outperform text-only baselines and rival explicit prompts while requiring less effort. The work highlights the feasibility and value of gaze-driven personalization for future mobile and wearable LLM applications and outlines privacy, edge-computation, and cognitive-state sensing as key future directions.

Abstract

Smart glasses are accelerating progress toward more seamless and personalized LLM-based assistance by integrating multimodal inputs. Yet, these inputs rely on obtrusive explicit prompts. The advent of gaze tracking on smart devices offers a unique opportunity to extract implicit user intent for personalization. This paper investigates whether LLMs can interpret user gaze for text-based tasks. We evaluate different gaze representations for personalization and validate their effectiveness in realistic reading tasks. Results show that LLMs can leverage gaze to generate high-quality personalized summaries and support users in downstream tasks, highlighting the feasibility and value of gaze-driven personalization for future mobile and wearable LLM applications.

Figures (9)

• Figure 1: By integrating gaze as a proxy for user attention, our approach enables large language models to produce summaries centered on the content users find most relevant to their interest.
• Figure 2: Generate personalized summary using different gaze representations.
• Figure 3: Implementation of different summarization methods. Refer to Appendix A for the details of prompts.
• Figure 4: (A) Gaze effectively guides LLMs to generate summaries that focus on the content users attend to. (B) At the lexical level, Heatmap outperforms other gaze representations and even surpasses explicit-input summaries (Target Paragraphs) at the phrase level. (C) Semantically, there is no significant difference. (D) Personalization does not significantly degrade summary quality.
• Figure 5: (A) During the user study, the eye tracker was fixed below the screen. (B) User interface for questions. (C) User interface for writing task.