Memory Reviver: Supporting Photo-Collection Reminiscence for People with Visual Impairment via a Proactive Chatbot

TL;DR

Memory Reviver tackles the challenge of enabling reminiscence with photo collections for people with visual impairment by introducing a proactive chatbot that uses a Memory Tree to organize information and a Proactive Strategy to guide conversations. The approach delivers a storyline-based overview, progressive scene details, and timely suggestions for new scenes, improving memory recall, understanding of photo collections, and conversational engagement. In a 12-participant study, Memory Reviver outperformed a naïve GPT-4V baseline in scene coverage, memory recall, and user satisfaction, while uncovering considerations around accuracy, personalization, and privacy. The work offers practical design implications for accessible chatbots and presents a blueprint for future improvements in cross-scene reminiscence, long-term memory, and on-device processing to enhance privacy and user control.

Abstract

Reminiscing with photo collections offers significant psychological benefits but poses challenges for people with visual impairment (PVI). Their current reliance on sighted help restricts the flexibility of this activity. In response, we explored using a chatbot in a preliminary study. We identified two primary challenges that hinder effective reminiscence with a chatbot: the scattering of information and a lack of proactive guidance. To address these limitations, we present Memory Reviver, a proactive chatbot that helps PVI reminisce with a photo collection through natural language communication. Memory Reviver incorporates two novel features: (1) a Memory Tree, which uses a hierarchical structure to organize the information in a photo collection; and (2) a Proactive Strategy, which actively delivers information to users at proper conversation rounds. Evaluation with twelve PVI demonstrated that Memory Reviver effectively facilitated engaging reminiscence, enhanced understanding of photo collections, and delivered natural conversational experiences. Based on our findings, we distill implications for supporting photo reminiscence and designing chatbots for PVI.

Figures (6)

• Figure 1: The pipeline of the naïve chatbot.
• Figure 2: The Memory Tree organizes information in a photo collection into a three-level structure.
• Figure 3: (a) The Proactive Strategy starts the conversation with a storyline and then guides users to relive each scene. Within each scene, it introduces the scene activity, progressively presents scene details, and suggests the next scene at proper conversation rounds. (b) Users can freely switch scenes using two natural language commands.
• Figure 4: (a) Examples of the multi-round conversations. (b) The pipeline for Memory Reviver to generate a reply in each round.
• Figure 5: Task Performances in the evaluation study. Paired t-test was used for significance analysis.