AiGet: Transforming Everyday Moments into Hidden Knowledge Discovery with AI Assistance on Smart Glasses

TL;DR

AiGet tackles the decline of informal learning in daily life by introducing a proactive wearable AI assistant on AR smart glasses that analyzes real-time gaze, environmental context, and user profiles to deliver context-aware knowledge with minimal disruption. The approach combines a five-stage AiGet pipeline with mixed-initiative interactions, multimodal outputs, and personalized prompts, validated through in-lab ablation studies and real-world usage across diverse scenarios. Key contributions include design guidelines for context-aware informal learning, empirical evidence of AiGet's ability to surface unseen and unknown knowledge, and demonstrations of enhanced engagement and curiosity without compromising primary tasks. The work suggests practical potential for ubiquitous, personalized knowledge discovery in everyday environments and provides open-source resources to foster future development and deployment.

Abstract

Unlike the free exploration of childhood, the demands of daily life reduce our motivation to explore our surroundings, leading to missed opportunities for informal learning. Traditional tools for knowledge acquisition are reactive, relying on user initiative and limiting their ability to uncover hidden interests. Through formative studies, we introduce AiGet, a proactive AI assistant integrated with AR smart glasses, designed to seamlessly embed informal learning into low-demand daily activities (e.g., casual walking and shopping). AiGet analyzes real-time user gaze patterns, environmental context, and user profiles, leveraging large language models to deliver personalized, context-aware knowledge with low disruption to primary tasks. In-lab evaluations and real-world testing, including continued use over multiple days, demonstrate AiGet's effectiveness in uncovering overlooked yet surprising interests, enhancing primary task enjoyment, reviving curiosity, and deepening connections with the environment. We further propose design guidelines for AI-assisted informal learning, focused on transforming everyday moments into enriching learning experiences.

Figures (15)

• Figure 1: A day with AiGet, a wearable knowledge discovery assistant equipped with AR smart glasses and a ring mouse. (1) While casually walking, AiGet actively analyzes the user's gaze patterns and identifies "interesting" environmental entities. (2) AiGet recognizes the spider lily that the user briefly glanced at as a valuable opportunity for informal learning, considering her interest in plants in the User Profile and lack of knowledge about the flower. (3) Using contextual data, the system generates multiple knowledge candidates and selects the most valuable one about the spider lily for the user. (4) AiGet delivers multimodal feedback, including audio, text keywords with emojis, and an image with a bounding box for the flower to help the user locate the flower and gain information with minimal interference to primary tasks. (5) Surprised by the knowledge delivery about the overlooked flower, the user uses the ring mouse to ask follow-up questions. (6) Later, AiGet continuously helps her uncover design details behind the campus architecture and compare two unfamiliar snacks, aligning with her interest in healthy diets, etc. (7) When the user meets a friend who owns a cat, she shares what she learned from AiGet---that spider lilies are toxic to pets---enriching their social interaction.
• Figure 2: Proposed Framework for Generating Desired Knowledge from Daily Moments. Note: Gray dashed boxes and lines represent user suggestions for future system improvements, collected from the final real-world study (Sec \ref{['sec:discussion:future_improvements']}) and not implemented or evaluated in the current paper. Our implementation, AiGet, is detailed in system design (Sec \ref{['sec:system']}).
• Figure 3: AiGet's System Processing Pipeline. (1) LLM Request Trigger: Supports mixed-initiative knowledge queries. (2) Context Analysis: Analyzes real-time user attention and long-term interests and infers learning desires. (3) Knowledge Generation & Prioritization: Filters redundant content, prioritizing the most valuable knowledge to avoid overload. (4) Output Transformation: Presents knowledge in a multimodal format to balance engagement and cognitive load. (5) Follow-Up User Actions: Enables user control over interaction. Some details are omitted in the figure. Full details and agent prompts are in Appendix \ref{['appendix:prompt_for_llm']}.
• Figure 4: An example of comparative knowledge from three MLLM pipelines. Note: Participants can't see each pipeline's name (in purple).
• Figure 5: Subjective ratings evaluating the desirability of generated knowledge. * indicates significance of $p<0.05$, ** indicates significance of $p<0.01$, *** indicates significance of $p<0.001$.