Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship

TL;DR

The paper tackles disruptions that break immersion in immersive VR work by enabling cross-domain observation of both physical and virtual states. It proposes NEO, a neighbor-environment observer with perception–decision–action modules and a joint observation pipeline that fuses PE, PU, VE, and VU data via an And-Or Graph-based decision module, with the probabilistic model $P(A_t, C_t, pt_t) = P(A_t|C_t, pt_t) P(C_t|pt_t) P(pt_t)$. A personalization mechanism uses a sigmoid-based update to $P(A|C, pt)$ based on user feedback. Experiments and a user study show that NEO reduces workload, increases engagement, and is adaptable to different sensor configurations and user preferences, with promising applications in smart-home and beyond.

Abstract

Human-computer symbiosis is a crucial direction for the development of artificial intelligence. As intelligent systems become increasingly prevalent in our work and personal lives, it is important to develop strategies to support users across physical and virtual environments. While technological advances in personal digital devices, such as personal computers and virtual reality devices, can provide immersive experiences, they can also disrupt users' awareness of their surroundings and enhance the frustration caused by disturbances. In this paper, we propose a joint observation strategy for artificial agents to support users across virtual and physical environments. We introduce a prototype system, neighbor-environment observer (NEO), that utilizes non-invasive sensors to assist users in dealing with disruptions to their immersive experience. System experiments evaluate NEO from different perspectives and demonstrate the effectiveness of the joint observation strategy. A user study is conducted to evaluate its usability. The results show that NEO could lessen users' workload with the learned user preference. We suggest that the proposed strategy can be applied to various smart home scenarios.

Figures (14)

• Figure 1: The NEO system jointly observes the physical and virtual environments and takes action in the two environments simultaneously. It is designed to deal with disruptions for people when they are immersed in a virtual working environment.
• Figure 2: External stimuli and disruptions may break the immersion. We propose an intelligent agent NEO that could perceive user's demands, help them to handle the disruptions, and preserve their immersion in virtual environments.
• Figure 3: Information and decision transfer process in our framework. Joint observation information is transmitted to a data manager via WiFi. The data manager activates the decision manager, from which the decision is transmitted to the two types of embodiment.
• Figure 4: The And-Or Graph of the joint observation. The perception of the agent is categorized into two sets: user state and environment state. The user state node consists of four components, representing occupation methods, which are further divided based on method classes and specific methods. The environment state node consists of two components: the virtual part and the physical part. Colors of the Terminal-nodes denote whether the nodes belong to the physical environment or not.
• Figure 5: The agent adjusts the conditional probability along the sigmoid curve according to the user's feedback. It stops when reaching the upper or lower bound.