Bridging the Awareness Gap: Socially Mediated State Externalization for Transparent Distributed Home Robots

Abstract

Distributed multi-robot systems for the home often require robots to operate out of the user's sight, creating a state awareness gap that can diminish trust and perceived transparency and control. This paper investigates whether real-time, socially mediated state externalization can bridge this gap without compromising task performance. We developed a system where a co-located social mediator robot (Pepper) externalizes the hidden execution states of an out-of-sight mobile manipulator (Stretch~3) for voice-driven object retrieval and delivery, where task-level states are synchronized and externalized through verbal updates and visual progress display. In a counterbalanced within-subject study (N=30), we compared a baseline of Autonomous Hidden Execution against Socially Mediated State Externalization. Our results show that externalization significantly increases user task-focused attention (from 15.8% to 84.6%, p<.001) and substantially improves perceived perspicuity, dependability, stimulation, and attractiveness (all p<.001). Furthermore, 83% of participants preferred the externalized condition, and this improvement in user experience was achieved without a statistically significant increase in end-to-end task completion time (p=.271). The results suggest that socially mediated state externalization is an effective architectural mechanism for designing more transparent and trustworthy distributed robot systems, ultimately enhancing user experience without sacrificing performance in distributed home robot deployments.

Figures (7)

• Figure 2: Overview of the distributed execution architecture and state externalization framework. The user interacts with a co-located social mediator, while task execution is performed by a mobile manipulation robot operating in a physically separated workspace. A coordination server synchronizes task-level execution states (e.g., NAVIGATING, GRASPING, RECOVERING) and externalizes them to the user through verbal and visual feedback. This architecture enables real-time state awareness despite spatial separation.
• Figure 3: System architecture of the distributed robot interaction framework. User voice commands are captured by the Pepper social mediator, which forwards structured intents to a coordination server. The server dispatches task commands to the Stretch mobile manipulator for navigation and object retrieval. During execution, the robot publishes task-level state updates (e.g., NAVIGATING, SEARCHING, GRASPING), which are externalized to the user through Pepper via verbal and visual feedback. When a failure is detected, a closed-loop recovery process is triggered: the failure event is externalized to the user, user confirmation is collected, and a recovery command is issued to the execution robot.
• Figure 4: Experimental layout of the distributed task execution setup under two conditions. (a) Condition A: Autonomous Hidden Execution: the execution robot operates in a physically separated workspace without communicating intermediate task states to the user. (b) Condition B: Socially Mediated State Externalization: a co-located social mediator provides real-time verbal and visual updates about the execution robot’s task states. In both conditions, the execution robot performs object retrieval in a separate room, while the user remains spatially separated from task execution. The purple arrow indicates the execution robot’s navigation path during retrieval and delivery tasks.
• Figure 5: Example end-to-end trial timeline for Condition B (Socially Mediated State Externalization). (a) The participant issues a request via the social mediator (Pepper) and confirms the task. (b) While the execution robot operates in a separate workspace, the mediator externalizes task progress through verbal updates and a visual progress display. (c) The execution robot retrieves the target object and returns. (d) The item is delivered to the participant, and the mediator concludes the interaction.
• Figure 6: Objective performance and interaction metrics under Hidden Execution and Socially Mediated State Externalization. (a) Time-based metrics, including task initiation time, execution time, and total end-to-end duration. (b) Average number of grasp attempts per task session. (c) Task-focused attention ratio during execution. Error bars represent $\pm$1 SEM. Asterisks denote statistical significance ($^*p<0.05$, $^{***}p<0.001$)..