Eye2Recall: Exploring the Design of Enhancing Reminiscence Activities via Eye Tracking-Based LLM-Powered Interaction Experience for Older Adults

TL;DR

Eye2Recall investigates enhancing photo-based reminiscence for older adults by fusing gaze signals with an LLM-powered conversational agent. Through expert interviews (N=4) and a user study (N=12), the authors design and evaluate a gaze-aware reminiscence prototype that adapts dialogue around visual ROIs, using two prompts per photo to balance summary and targeted questions. Findings show high usability, low workload, and positive affect, with gaze-informed prompts yielding more personalized, emotionally meaningful conversations and deeper reflection. The work contributes design guidelines for safety-by-design, culturally aware content, and robust multimodal interaction, highlighting practical implications for scalable, accessible reminiscence technologies that align with older adults’ natural interaction patterns.

Abstract

Photo-based reminiscence has the potential to have a positive impact on older adults' reconnection with their personal history and improve their well-being. Supporting reminiscence in older adults through technological implementations is becoming an increasingly important area of research in the fields of HCI and CSCW. However, the impact of integrating gaze and speech as mixed-initiative interactions in LLM-powered reminiscence conversations remains under-explored. To address this, we conducted expert interviews to understand the challenges that older adults face with LLM-powered, photo-based reminiscence experiences. Based on these design considerations, we developed Eye2Recall, a system that integrates eye tracking for detecting visual interest with natural language interaction to create a mixed-initiative reminiscence experience. We evaluated its effectiveness through a user study involving ten older adults. The results have important implications for the future design of more accessible and empowering reminiscence technologies that better align with older adults' natural interaction patterns and enhance their positive aging.

Figures (11)

• Figure 1: Participants use the Eye2Recall prototype to interact with the LLM-powered agent.
• Figure 2: System pipeline of the Eye2Recall prototype. The system includes (I) a visual exploration module with eye-tracking calibration, gaze recording, and attention/ROI detection, and (II) a conversational interaction module comprising external-cue--guided storytelling and autobiographical memory activation. Steps 2--5 repeat for each user query in our LLM-powered pipeline.
• Figure 3: (a) Example of visual content categorization, consisting of generic old photos categorized into five themes. (b) Screenshots of the Eye2Recall user interface (UI).
• Figure 4: Prompt task instructions and Chat History in Eye2Recall. On the left side, there are descriptions outlining the overall task along with detailed instructions for two distinct modules. On the right side, an example conversation between an LLM-powered agent and a user is provided.
• Figure 5: The user study process comprised three phases: (1) the introduction and pre-evaluation, (2) the prototype testing, and (3) semi-structured interviews with the post-evaluation. Each participant participated for approximately 1.5 hours.