Sampling-Based Motion Planning with Scene Graphs Under Perception Constraints

TL;DR

MOPS-PRM is proposed, a roadmap-based motion planner that integrates the perception cost of observing multiple objects or humans directly into motion planning for high-DoF robots, and is extensively validated in both simulated and real-world experiments.

Abstract

It will be increasingly common for robots to operate in cluttered human-centered environments such as homes, workplaces, and hospitals, where the robot is often tasked to maintain perception constraints, such as monitoring people or multiple objects, for safety and reliability while executing its task. However, existing perception-aware approaches typically focus on low-degree-of-freedom (DoF) systems or only consider a single object in the context of high-DoF robots. This motivates us to consider the problem of perception-aware motion planning for high-DoF robots that accounts for multi-object monitoring constraints. We employ a scene graph representation of the environment, offering a great potential for incorporating long-horizon task and motion planning thanks to its rich semantic and spatial information. However, it does not capture perception-constrained information, such as the viewpoints the user prefers. To address these challenges, we propose MOPS-PRM, a roadmap-based motion planner, that integrates the perception cost of observing multiple objects or humans directly into motion planning for high-DoF robots. The perception cost is embedded to each object as part of a scene graph, and used to selectively sample configurations for roadmap construction, implicitly enforcing the perception constraints. Our method is extensively validated in both simulated and real-world experiments, achieving more than ~36% improvement in the average number of detected objects and ~17% better track rate against other perception-constrained baselines, with comparable planning times and path lengths.

Figures (5)

• Figure 1: An illustration of our perception-aware motion planner that leverages a scene graph embedded with perception costs to generate a trajectory from a start to a goal, while monitoring three objects of interest. The screen of the monitor is the preferable viewpoint in this scenario.
• Figure 2: This figure presents the pipeline of our planner. The planner takes the scene graph as input, combining geometric and object-level information with a neural perception cost function to perform multi-object constrained sampling. Sampling is performed (see \ref{['sec:method_multiobject']}) to construct a PRM, which is searched using A* to generate a trajectory that effectively accomplishes perception tasks involving multiple objects along the path.
• Figure 3: In our simulated benchmarks, the robot moves from a start to a goal in an office environment while monitoring the four screens of the monitors placed on the table. The robot takes the longer path to observe the monitors, where the arrows illustrate the camera orientations. The bottom plot shows the camera pan-tilt joint angles along the trajectory.
• Figure 4: In this real-robot experiment, the robot plans two different paths with the same start (shown in red) and goal (shown in green) based on a user-specified importance of the two paintings: the yellow path prioritizes painting 1, while the blue path prioritizes painting 2. Both paths start by observing a human with a monitor and end by looking at another human sitting at the table while the middle sections of the paths differ as they prioritize observing different paintings.
• Figure 5: Performance of MOPS-PRM under varying number of objects and PRM sizes. In the first column, the number of objects is fixed at five. In the second, PRM size is approximately 300 nodes. We report the planning and PRM construction times, and the average number of detections per frame.