AR Secretary Agent: Real-time Memory Augmentation via LLM-powered Augmented Reality Glasses

TL;DR

The paper introduces AR Secretary Agent, an external memory augmentation system that uses AR glasses, audio capture, and LLM-powered summarization to identify speakers and recap prior interactions in real time. It details a modular design combining minimalist AR UI, Whisper-based transcription, GPT-4-generated summaries, and face-recognition-driven retrieval, with data stored in an SQL backend and accessible via a web app. A user study with twelve participants demonstrates memory improvements in short- and long-term recall, along with high usability ratings, while also highlighting limitations in microphone quality, multilingual capabilities, and multi-speaker scenarios. The work advances practical memory augmentation for information-heavy settings and outlines concrete paths for making AR-based memory aids more robust, discreet, and scalable. Overall, the AR Secretary demonstrates the potential of integrating LLMs with wearable AR interfaces to support memory, social interaction, and professional workflows, while signaling hardware and privacy considerations for broader adoption.

Abstract

Interacting with a significant number of individuals on a daily basis is commonplace for many professionals, which can lead to challenges in recalling specific details: Who is this person? What did we talk about last time? The advant of augmented reality (AR) glasses, equipped with visual and auditory data capture capabilities, presents a solution. In our work, we implemented an AR Secretary Agent with advanced Large Language Models (LLMs) and Computer Vision technologies. This system could discreetly provide real-time information to the wearer, identifying who they are conversing with and summarizing previous discussions. To verify AR Secretary, we conducted a user study with 13 participants and showed that our technique can efficiently help users to memorize events by up to 20\% memory enhancement on our study.

Figures (10)

• Figure 1: Augmented Reality Interface
• Figure 2: Detailed Pipeline
• Figure 3: Box-plot and Wilcoxon test (** $p < 0.01$) on overall averaged percentage of speech remembered, without and with help of AR assistant, for short term-memory
• Figure 4: Box-plot and Wilcoxon test (*** $p < 0.001$) on averaged percentage of speech remembered, for every individual speaker, without and with help of AR assistant, for short term-memory
• Figure 5: Box-plot and Wilcoxon test (ns $p > 0.05$) on overall averaged percentage of speech remembered, for every memory baseline, without and with help of AR assistant, for short term-memory