Effective Task Planning with Missing Objects using Learning-Informed Object Search

TL;DR

This work develops a planning framework centered around novel model-based LIOS actions: each a policy that aims to find and retrieve a single object for effective, sound, and complete learning-informed task planning despite uncertainty.

Abstract

Task planning for mobile robots often assumes full environment knowledge and so popular approaches, like planning via the PDDL, cannot plan when the locations of task-critical objects are unknown. Recent learning-driven object search approaches are effective, but operate as standalone tools and so are not straightforwardly incorporated into full task planners, which must additionally determine both what objects are necessary and when in the plan they should be sought out. To address this limitation, we develop a planning framework centered around novel model-based LIOS actions: each a policy that aims to find and retrieve a single object. High-level planning treats LIOS actions as deterministic and so -- informed by model-based calculations of the expected cost of each -- generates plans that interleave search and execution for effective, sound, and complete learning-informed task planning despite uncertainty. Our work effectively reasons about uncertainty while maintaining compatibility with existing full-knowledge solvers. In simulated ProcTHOR homes and in the real world, our approach outperforms non-learned and learned baselines on tasks including retrieval and meal prep.

Figures (8)

• Figure 1: Overview: Learning-Informed Task Planning with Unknown Object Locations. Standard task planners, like (left), typically assume full environment knowledge and cannot plan when objects are missing. Our approach (middle) adds find: an action to find and retrieve missing objects that high-level planning treats as deterministic. find actions (right) are instantiated via learning-informed object search policies, whose expected costs inform high-level planning.
• Figure 2: Toy scenario showing computation of expected cost $Q^{\pi_o}$ for an example policy via Eq. \ref{['eq:lsp']}. In experiments, learning informs $P_{\textsc{\tiny{}found}}{}$.
• Figure 3: A. The Boston Dynamics Spot robot, equipped with a Spot Arm for manipulation and a LiDAR sensor payload for mapping and localization. B. A top-down 3D map of the indoor home environment, showing the cluttered layout with furniture and the robot's position.
• Figure 4: Object search performance results. Each point in the scatter plot (left) is a single trial to find an object. Results show the improvement of our policy versus a Greedy baseline. The example trial (right) shows trajectories from each strategy.
• Figure 5: Results: Breakfast Prep Scenario, in which the robot is tasked to prepare and serve breakfast. We include an annotated example trial (left) and scatter plots (right) that show the per-trial performance of our ModelLIOS strategy versus each baseline. Our approach effectively completes complex tasks with multiple solution pathways despite missing objects.