On the Emergence of Symmetrical Reality

TL;DR

This work proposes symmetrical reality (SR), a unified framework that integrates physical and virtual worlds via dual perceptual centers: human minds in the physical world and AI minds in the virtual world. It argues that existing VR/AR/MR approaches neglect AI-centered perception and that SR enables symmetric perception and bidirectional interaction, enabling sustained human-AI coexistence. Contributions include a formal SR framework, analysis of technical challenges, a pouring-water prototype to illustrate bidirectional interaction, and two exemplar system designs plus an active assisting service. The study provides a roadmap for researchers and practitioners to design robust, human-compatible AI within interconnected physical-virtual ecosystems, with implications for XR, AI safety, and future AGI deployment.

Abstract

Artificial intelligence (AI) has revolutionized human cognitive abilities and facilitated the development of new AI entities capable of interacting with humans in both physical and virtual environments. Despite the existence of virtual reality, mixed reality, and augmented reality for several years, integrating these technical fields remains a formidable challenge due to their disparate application directions. The advent of AI agents, capable of autonomous perception and action, further compounds this issue by exposing the limitations of traditional human-centered research approaches. It is imperative to establish a comprehensive framework that accommodates the dual perceptual centers of humans and AI agents in both physical and virtual worlds. In this paper, we introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. This framework enables researchers to better comprehend how AI agents can collaborate with humans and how distinct technical pathways of physical-virtual integration can be consolidated from a broader perspective. We then delve into the coexistence of humans and AI, demonstrating a prototype system that exemplifies the operation of symmetrical reality systems for specific tasks, such as pouring water. Subsequently, we propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality. This paper aims to offer beneficial perspectives and guidance for researchers and practitioners in different fields, thus contributing to the ongoing research about human-AI coexistence in both physical and virtual environments.

Figures (10)

• Figure 1: The topological structure of symmetrical reality. Symmetrical reality integrates the physical and virtual worlds into a unified system. Humans inhabit the physical world and possess virtual embodiments within the virtual domain, while AI agents originate in the virtual world and maintain physical embodiments in the physical world. The symmetrical attributes of this system are manifested through symmetrical perception and bidirectional interaction between humans and AI agents.
• Figure 2: The cognition continuum. (1) The top row represents the virtuality continuum proposed by Milgram et al.milgram1994taxonomymilgram1995augmented. (2) The bottom row depicts the cognition continuum of humans and AI. Analogous to the definition of Mixed Reality, Mixed Cognition refers to the proportion of human and AI cognition within a single system.
• Figure 3: Topological symmetry of symmetrical reality. Humans, as components of the physical world, are capable of perceiving both the physical and virtual worlds, including themselves. Similarly, AI agents, as components of the virtual world, are capable of perceiving both the physical and virtual worlds, including themselves. The physical and virtual worlds are heterogeneous with respect to their forms of existence, while entities within the same world are homogeneous.
• Figure 4: The reality framework based on the combination of the virtuality continuum and the cognition continuum.
• Figure 5: Symmetrical reality and other special forms. The top figure illustrates symmetrical reality, while the smaller figures at the bottom depict six distinct special cases, i.e., physical reality (PR), augmented reality (AR), virtual reality (VR), pure virtuality (PV), inverse augmented reality (IAR), and inverse virtual reality (IVR). According to the Reality-Virtuality Continuum proposed by Milgram et al.milgram1994taxonomymilgram1995augmentedmilgram1999taxonomy, mixed reality (MR) comprises AR and Augmented Virtuality (AV). For the sake of convenience, this paper will use AR as a representative of all physical-virtual mixed frameworks.