Integrating Field of View in Human-Aware Collaborative Planning

TL;DR

The paper tackles human field-of-view limitations in human–robot collaboration by embedding FOV-aware knowledge updates into a POMDP framework and deploying a hierarchical online planner that operates on abstract states to manage large state spaces. By simulating and validating in a Steakhouse domain and a VR kitchen, it demonstrates that guiding robot actions to remain within the human's perception window reduces KB gaps and interruptions without sacrificing task progress. The key contribution is a scalable, real-time planning approach that updates the humanKB as objects enter the FOV, enabling proactive, non-intrusive communication through action placement and movement. The findings offer practical implications for designing collaborative robots that adapt to human perceptual constraints in dynamic, fast-paced tasks.

Abstract

In human-robot collaboration (HRC), it is crucial for robot agents to consider humans' knowledge of their surroundings. In reality, humans possess a narrow field of view (FOV), limiting their perception. However, research on HRC often overlooks this aspect and presumes an omniscient human collaborator. Our study addresses the challenge of adapting to the evolving subtask intent of humans while accounting for their limited FOV. We integrate FOV within the human-aware probabilistic planning framework. To account for large state spaces due to considering FOV, we propose a hierarchical online planner that efficiently finds approximate solutions while enabling the robot to explore low-level action trajectories that enter the human FOV, influencing their intended subtask. Through user study with our adapted cooking domain, we demonstrate our FOV-aware planner reduces human's interruptions and redundant actions during collaboration by adapting to human perception limitations. We extend these findings to a virtual reality kitchen environment, where we observe similar collaborative behaviors.

Figures (7)

• Figure 1: Hierarchical online planning. We start by rolling out an action $a$ and obtaining new states (left). Each new state $s$ undergoes random exploration to obtain observations $o$ (center). Each rolled-out state (marked in orange), with a KB not seen in previously explored states, is reduced to an abstract state $s^A$. We then perform a look-ahead in the abstract state space (right). The costs between abstract states, shown in the lower graph, are computed by mapping abstract states back to the original state space. The planner ultimately chooses the action from the first stage that results in the highest value $V(s')$ (highlighted in green), indicating the optimal action.
• Figure 2: Steakhouse domain. (Dimmed tiles are completely black during user studies to simulate FOV.)
• Figure 3: Collaboration behavior in a Peninsula kitchen. Top row: the baseline robot placing an onion directly on the chopping board; Bottom row: the FOV-aware robot revealing the onion to the human before proceeding. (Dimmed tiles are black during user studies to simulate FOV.)
• Figure 4: Collaboration behavior in a $\cap$-shaped kitchen. Top row: the baseline robot (green agent) picks up a plate assuming the human is aware. Bottom row: the FOV-aware robot waits for the human to pick up the cooked steak. (Dimmed tiles are black during the study to simulate FOV.)
• Figure 5: The FOV-aware robot (green agent) follows a staircase-like trajectory to remain within the human FOV.