Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistant

TL;DR

To address the lack of long-term, personalized context in AR assistants, the authors introduce Memento, a proactive memory-aware AR system. Memento lifelogs user verbal queries as Referent-anchored Spatiotemporal Activity Memories (RSAM) and retrieves them via a hybrid R-tree plus HNSW index to surface contextually relevant memories in daily settings. The pipeline combines egocentric sensing, open-vocabulary referent detection, CLIP embeddings, and LLM reasoning to generate and locate proactive visualizations anchored to real-world referents. Preliminary technical and user evaluations reveal encouraging accuracy and acceptance, while outlining privacy, usability, and hardware limitations as key areas for future work.

Abstract

We introduce Memento, a conversational AR assistant that permanently captures and memorizes user's verbal queries alongside their spatiotemporal and activity contexts. By storing these "memories," Memento discovers connections between users' recurring interests and the contexts that trigger them. Upon detection of similar or identical spatiotemporal activity, Memento proactively recalls user interests and delivers up-to-date responses through AR, seamlessly integrating AR experience into their daily routine. Unlike prior work, each interaction in Memento is not a transient event, but a connected series of interactions with coherent long--term perspective, tailored to the user's broader multimodal (visual, spatial, temporal, and embodied) context. We conduct preliminary evaluation through user feedbacks with participants of diverse expertise in immersive apps, and explore the value of proactive context-aware AR assistant in everyday settings. We share our findings and challenges in designing a proactive, context-aware AR system.

Figures (4)

• Figure 1: The view of situated AR in Memento. A physical referent derives its meaning from its spatiotemporal context (Where and When of the object). AR content anchored to it serves as an augmentation layer that extends the physically visible information of the referent, within the same context. The referent acts as the bridge between the physical and virtual worlds.
• Figure 2: Memento pipeline overview: With a verbal query, the visual, head-gaze, and the transcribed textual query from the user are sent to Memento. Upon completion of response generation, the answer is situated onto the referent that the query is initiated to, and is permanently stored along with its spatial activity context as a form of a memory. After the initial query, the memory is proactively recalled when the user's similar contextual spatial activity, and situated onto the referent of the user's prior interest, without an explicit query from the user. \ref{['subsubsec:personal_assistant']}, \ref{['subsubsec:spatial_activity_recognizer']}, \ref{['subsubsec:memory_generation']}, \ref{['subsubsec:memory_retrieval']}, \ref{['subsubsec:proactive_visualization']}, \ref{['subsubsec:adjustment_inferface']}
• Figure 3: User interfaces of Memento. (A) Example visualization of RSAM situated onto a 'Water bottle' referent. (B) Context adjustment interface provides a switch between Query (New query) and Proactive (Prior query/Memory recall) mode, and adjust the interval of memory recalls (How often memory is recalled).
• Figure 4: The three real-world use-cases of Memento. Upon the contextual alignment of the user's current spatial activity (Referent, Space, Time, Activity), Memento proactively recalls a prior RSAM and situates via AR.