Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmenting Human Cognition through Everyday AR

Xiaoan Liu

TL;DR

This paper addresses the challenge of transforming augmented reality from a passive overlay into a cognitive augmentation layer embedded in physical environments. It proposes reality computing with an always-on semantic layer that integrates gaze, speech, gestures, and multimodal large language models to provide proactive, context-aware guidance. Key contributions include the conceptual framing of reality as a canvas, the semantic layer for natural and contextual interaction, and demonstrations like the RealiTips prototype and in-scene task guidance across scenarios such as device pairing, library knowledge discovery, and Arduino assembly. The work highlights the potential of AR to reduce cognitive overhead, accelerate learning, and extend everyday cognition by tightly coupling digital intelligence with physical surfaces and actions.

Abstract

As spatial computing and multimodal LLMs mature, AR is tending to become an intuitive "thinking tool," embedding semantic and context-aware intelligence directly into everyday environments. This paper explores how always-on AR can seamlessly bridge digital cognition and physical affordances, enabling proactive, context-sensitive interactions that enhance human task performance and understanding.

Augmenting Human Cognition through Everyday AR

TL;DR

This paper addresses the challenge of transforming augmented reality from a passive overlay into a cognitive augmentation layer embedded in physical environments. It proposes reality computing with an always-on semantic layer that integrates gaze, speech, gestures, and multimodal large language models to provide proactive, context-aware guidance. Key contributions include the conceptual framing of reality as a canvas, the semantic layer for natural and contextual interaction, and demonstrations like the RealiTips prototype and in-scene task guidance across scenarios such as device pairing, library knowledge discovery, and Arduino assembly. The work highlights the potential of AR to reduce cognitive overhead, accelerate learning, and extend everyday cognition by tightly coupling digital intelligence with physical surfaces and actions.

Abstract

As spatial computing and multimodal LLMs mature, AR is tending to become an intuitive "thinking tool," embedding semantic and context-aware intelligence directly into everyday environments. This paper explores how always-on AR can seamlessly bridge digital cognition and physical affordances, enabling proactive, context-sensitive interactions that enhance human task performance and understanding.
Paper Structure (9 sections, 4 figures)

This paper contains 9 sections, 4 figures.

Table of Contents

  1. INTRODUCTION
  2. REALITY AS THE CANVAS
  3. AR as an Always-On Knowledge Layer
  4. Moving from "Screens" to "Scenes
  5. THE SEMANTIC LAYER FOR REALITY
  6. Natural and Contextual Interaction
  7. Proactive Thinking Tools
  8. LEVERAGING THE AFFORDANCES OF THE PHYSICAL WORLD
  9. CONCLUSION

Figures (4)

  • Figure 1: A scenario in AR where a MacBook and AirPods on a desk are labeled, with a 'Pair' prompt suggesting their connection, allowing users to initiate pairing with a gesture by directly manipulating the virtual connector.
  • Figure 2: RealiTips providing step-by-step guidance in AR.
  • Figure 3: A library scenario showing AI-assisted semantic search. The AR system recognizes multiple books, such as Software Testing Foundations and Seven Databases in Seven Weeks, allowing the user to ask for summaries, related topics, or best practices in context.
  • Figure 4: Step-by-step instructions for assembling an Arduino-based robotic platform. The AR system tracks the user's progress and displays targeted prompts (e.g., "Attach the V5 Expanding Board"), reducing errors and accelerating learning.