Learning Orbitally Stable Systems for Diagrammatically Teaching

TL;DR

SDDT addresses the problem of teaching robots to approach a surface and execute user-defined cyclic motions from a single 2D sketch. It achieves this by morphing a known Orbitally Asymptotically Stable base system into a target shape using a parameterized diffeomorphism implemented with an invertible neural network, with the target cycle constrained to the sketch via a ray-traced projection and a Hausdorff distance loss. The authors provide theoretical guarantees that any smooth closed 2D curve is morphable to the base cycle and show empirical success in simulation and on real hardware, outperforming neural ODE baselines and a static base system. The approach is particularly suited to mobile manipulators with egocentric vision for sketch-based diagrammatic teaching, enabling robust, complex cyclic tasks like painting, wiping, or sanding with minimal user input.

Abstract

Diagrammatic Teaching is a paradigm for robots to acquire novel skills, whereby the user provides 2D sketches over images of the scene to shape the robot's motion. In this work, we tackle the problem of teaching a robot to approach a surface and then follow cyclic motion on it, where the cycle of the motion can be arbitrarily specified by a single user-provided sketch over an image from the robot's camera. Accordingly, we contribute the Stable Diffeomorphic Diagrammatic Teaching (SDDT) framework. SDDT models the robot's motion as an Orbitally Asymptotically Stable (O.A.S.) dynamical system that learns to stablize based on a single diagrammatic sketch provided by the user. This is achieved by applying a \emph{diffeomorphism}, i.e. a differentiable and invertible function, to morph a known O.A.S. system. The parameterised diffeomorphism is then optimised with respect to the Hausdorff distance between the limit cycle of our modelled system and the sketch, to produce the desired robot motion. We provide novel theoretical insight into the behaviour of the optimised system and also empirically evaluate SDDT, both in simulation and on a quadruped with a mounted 6-DOF manipulator. Results show that we can diagrammatically teach complex cyclic motion patterns with a high degree of accuracy.

Figures (10)

• Figure 1: Diagrammatic teaching is a paradigm to interface with robots by drawing sketches over camera images. We contribute SDDT to diagrammatically teach robots robot policies that approach a surface in view and stabilise at cyclic motions of the provided shape on the surface. (Left) A sketch of the desired pentagon-shaped cycle (in red) is provided by the user from the egocentric view of the robot. (Right) The resulting policy forces the end-effector to quickly approach the surface, and then stabilise to continuously trace out the shape of the provided sketch.
• Figure 2: Diffeomorphisms can be thought of as "morphing" a dynamical system into one another. (Left) Five trajectories (red) of overlaid on grid points (blue); (Right) Morphed trajectories and the corresponding grid.
• Figure 3: Trajectories converge to a limit cycle at $y=0$.
• Figure 4: We visualise how the ambient space is morphed by the learnt diffeomorphism: (Left) Transport map with points on the base system (blue) mapped (shown by grey line) onto those of the learned system (red); (Right) Concentric circles passed through the diffeomorphism to match the desired shape.
• Figure 5: Qualitative results of learning diffeomorphisms to shape the circular limit cycle into outlines of the whale, dog, flower, and eagle. We also show, at different viewing angles, of example trajectories integrated from multiple initial 3D positions (in green). We observe that each trajectory is able to converge onto the shaped limit cycle on the surface.

Theorems & Definitions (3)

• Definition III.1: O.A.S. Stability
• Proposition IV.1
• proof