Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching Robots Where To Go And How To Act With Human Sketches via Spatial Diagrammatic Instructions

Qilin Sun, Weiming Zhi, Tianyi Zhang, Matthew Johnson-Roberson

TL;DR

It is demonstrated how Spatial Diagrammatic Instructions can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task.

Abstract

This paper introduces Spatial Diagrammatic Instructions (SDIs), an approach for human operators to specify objectives and constraints that are related to spatial regions in the working environment. Human operators are enabled to sketch out regions directly on camera images that correspond to the objectives and constraints. These sketches are projected to 3D spatial coordinates, and continuous Spatial Instruction Maps (SIMs) are learned upon them. These maps can then be integrated into optimization problems for tasks of robots. In particular, we demonstrate how Spatial Diagrammatic Instructions can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task. Human operators can specify, via sketch, spatial regions of interest for a manipulation task and permissible regions for the mobile manipulator to be at. Then, an optimization problem that maximizes the manipulator's reachability, or coverage, over the designated regions of interest while remaining in the permissible regions is solved. We provide extensive empirical evaluations, and show that our formulation of Spatial Instruction Maps provides accurate representations of user-specified diagrammatic instructions. Furthermore, we demonstrate that our diagrammatic approach to the Mobile Base Placement Problem enables higher quality solutions and faster runtime.

Teaching Robots Where To Go And How To Act With Human Sketches via Spatial Diagrammatic Instructions

TL;DR

It is demonstrated how Spatial Diagrammatic Instructions can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task.

Abstract

This paper introduces Spatial Diagrammatic Instructions (SDIs), an approach for human operators to specify objectives and constraints that are related to spatial regions in the working environment. Human operators are enabled to sketch out regions directly on camera images that correspond to the objectives and constraints. These sketches are projected to 3D spatial coordinates, and continuous Spatial Instruction Maps (SIMs) are learned upon them. These maps can then be integrated into optimization problems for tasks of robots. In particular, we demonstrate how Spatial Diagrammatic Instructions can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task. Human operators can specify, via sketch, spatial regions of interest for a manipulation task and permissible regions for the mobile manipulator to be at. Then, an optimization problem that maximizes the manipulator's reachability, or coverage, over the designated regions of interest while remaining in the permissible regions is solved. We provide extensive empirical evaluations, and show that our formulation of Spatial Instruction Maps provides accurate representations of user-specified diagrammatic instructions. Furthermore, we demonstrate that our diagrammatic approach to the Mobile Base Placement Problem enables higher quality solutions and faster runtime.
Paper Structure (15 sections, 13 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 15 sections, 13 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem Formulation
  4. Spatial Diagrammatic Instructions
  5. Mobile Base Placement Problem
  6. Optimization on Spatial Instruction Maps
  7. Projecting Spatial Diagrammatic Instructions into 3D
  8. Spatial Representations via Energy-Based Models
  9. Solving the MBPP
  10. Constraints
  11. Empirical Evaluation
  12. EBMs Represent SIMs Well
  13. Solving the MBPP with Spatial Diagrammatic Maps
  14. Multi-modality in Diagrammatic Instructions
  15. Conclusions and Future Work

Figures (4)

  • Figure 1: Spatial Diagrammatic Instructions enable users to sketch on camera images of the environment. Top: Red indicates regions where the user sketches as regions of interest, while green indicates regions the user sketches as permissible regions. Bottom: Continuous spatial representations of these sketched regions can then be incorporated as objectives and constraints within optimization problems to find optimal positions for mobile manipulator base placement.
  • Figure 2: Left: Examples of constraint SIMs. Contours indicate equal output values. Right: Examples of applying \ref{['alg:newtons-projection']} to project infeasible points (brown) back into the feasible region. Red arrows show the direction of projection. The resulting points on the boundary are shown in blue.
  • Figure 3: KDE, GMM, and our energy-based SIM fitted on different dataset of spatial points.
  • Figure 4: Row 1: Scenes with annotated SDIs. Row 2: SDIs projected into 3D. Row 3: Energy output levels from SIMs fitted on sketched ROIs. Row 4: Quadruped with manipulator posited at the optimal placements (indicated by green dot).