Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI

Sicheng Yang, Yukai Huang, Weitong Cai, Shitong Sun, Fengyi Fang, You He, Yiqiao Xie, Jiankang Deng, Hang Zhang, Jifei Song, Zhensong Zhang

TL;DR

Egocentric Co-Pilot is presented, a web-native neuro-symbolic framework that runs on smart glasses and uses a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools that support education, accessibility, and social inclusion for people who may benefit most from contextual, egocentric AI.

Abstract

What if accessing the web did not require a screen, a stable desk, or even free hands? For people navigating crowded cities, living with low vision, or experiencing cognitive overload, smart glasses coupled with AI agents could turn the web into an always-on assistive layer over daily life. We present Egocentric Co-Pilot, a web-native neuro-symbolic framework that runs on smart glasses and uses a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools. An egocentric reasoning core combines Temporal Chain-of-Thought with Hierarchical Context Compression to support long-horizon question answering and decision support over continuous first-person video, far beyond a single model's context window. Additionally, a lightweight multimodal intent layer maps noisy speech and gaze into structured commands. We further implement and evaluate a cloud-native WebRTC pipeline integrating streaming speech, video, and control messages into a unified channel for smart glasses and browsers. In parallel, we deploy an on-premise WebSocket baseline, exposing concrete trade-offs between local inference and cloud offloading in terms of latency, mobility, and resource use. Experiments on Egolife and HD-EPIC demonstrate competitive or state-of-the-art egocentric QA performance, and a human-in-the-loop study on smart glasses shows higher task completion and user satisfaction than leading commercial baselines. Taken together, these results indicate that web-connected egocentric co-pilots can be a practical path toward more accessible, context-aware assistance in everyday life. By grounding operation in web-native communication primitives and modular, auditable tool use, Egocentric Co-Pilot offers a concrete blueprint for assistive, always-on web agents that support education, accessibility, and social inclusion for people who may benefit most from contextual, egocentric AI.

Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI

TL;DR

Egocentric Co-Pilot is presented, a web-native neuro-symbolic framework that runs on smart glasses and uses a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools that support education, accessibility, and social inclusion for people who may benefit most from contextual, egocentric AI.

Abstract

What if accessing the web did not require a screen, a stable desk, or even free hands? For people navigating crowded cities, living with low vision, or experiencing cognitive overload, smart glasses coupled with AI agents could turn the web into an always-on assistive layer over daily life. We present Egocentric Co-Pilot, a web-native neuro-symbolic framework that runs on smart glasses and uses a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools. An egocentric reasoning core combines Temporal Chain-of-Thought with Hierarchical Context Compression to support long-horizon question answering and decision support over continuous first-person video, far beyond a single model's context window. Additionally, a lightweight multimodal intent layer maps noisy speech and gaze into structured commands. We further implement and evaluate a cloud-native WebRTC pipeline integrating streaming speech, video, and control messages into a unified channel for smart glasses and browsers. In parallel, we deploy an on-premise WebSocket baseline, exposing concrete trade-offs between local inference and cloud offloading in terms of latency, mobility, and resource use. Experiments on Egolife and HD-EPIC demonstrate competitive or state-of-the-art egocentric QA performance, and a human-in-the-loop study on smart glasses shows higher task completion and user satisfaction than leading commercial baselines. Taken together, these results indicate that web-connected egocentric co-pilots can be a practical path toward more accessible, context-aware assistance in everyday life. By grounding operation in web-native communication primitives and modular, auditable tool use, Egocentric Co-Pilot offers a concrete blueprint for assistive, always-on web agents that support education, accessibility, and social inclusion for people who may benefit most from contextual, egocentric AI.
Paper Structure (34 sections, 3 equations, 5 figures, 2 tables, 3 algorithms)

This paper contains 34 sections, 3 equations, 5 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Egocentric Artificial Intelligence.
  4. LLM-driven Agents and Tool Use.
  5. Neuro-Symbolic Systems.
  6. Multimodal Intent Disambiguation.
  7. Methodology
  8. Egocentric Reasoning Core
  9. LLM-Orchestrated Neuro-Symbolic Execution
  10. On-Device Perception and WebRTC-Based Interaction
  11. Proactive Multimodal Intent Disambiguation
  12. Runtime Guardrails and Schema Management.
  13. Experiments
  14. Application to Egocentric QA Benchmarks
  15. Ablation and Sensitivity Analysis
  16. ...and 19 more sections

Figures (5)

  • Figure 1: Our Reasoning Core pipeline. It integrates Temporal Chain-of-Thought (T-CoT) for short-term analysis and Hierarchical Context Compression (HCC) for long-term memory. The figure illustrates the T-CoT path where our fine-tuned MLLM processes a temporally bounded query.
  • Figure 2: On-device architecture for real-time multimodal interaction. Audio and video are parallel-processed and multiplexed into a bidirectional channel between smart glasses and the cloud backend. We use WebRTC DBLP:conf/mipro/SredojevSP15 (H.264 video, Opus audio, data channel), with a custom WebSocket variant as an on-premise baseline.
  • Figure 3: Core capabilities of the LLM-orchestrated framework. Our system interprets multimodal user intent and dynamically composes neuro-symbolic tools via the MCP protocol. (a) Foundational Tool Use: a simple query triggers a VLM for object recognition and an external API call. (b) Structured Task Management: natural language is translated into a structured API call for a native device application. (c) Complex Neuro-Symbolic Reasoning: the board-game co-pilot integrates a vision tool (neuro), a deterministic game engine (symbolic), and an LLM for semantic explanation. (d) Spatiotemporal Memory: the system resolves a deictic reference ("this") by visually tracking an object through occlusion and recalling it from memory.
  • Figure 4: Subjective evaluation of Egocentric Co-Pilot against commercial smart-glasses devices and a human baseline. Bars show mean 5-point Likert ratings (higher is better); asterisks (*) denote devices whose default interaction pattern deviates from continuous conversational AI.
  • Figure 5: Example interaction logs shown to participants. Each column corresponds to a different system's response to the same user query. By evaluating pre-recorded logs instead of live interactions, we avoid confounding AI quality with hardware, network, or UI differences.