Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching

TL;DR

This paper tackles learning robotic skills without kinesthetic or teleoperation by introducing Diagrammatic Teaching, where users provide 2D sketches on scene images to specify demonstrations. It then presents Ray-tracing Probabilistic Trajectory Learning (RPTL), which extracts time-varying view-space densities via normalizing flows, uses ray tracing to map densities to 3D regions, and fits a probabilistic end-effector trajectory model in 3D space. The approach supports adaptation to new starting poses and is validated through extensive simulation and real-world experiments, including intricate letter tracing and manipulation tasks on fixed-base and mobile manipulators. The results show that the generated 3D motions align with user sketches with high fidelity and robustness, illustrating a practical and scalable alternative to kinesthetic and teleoperation-based LfD.

Abstract

Learning for Demonstration (LfD) enables robots to acquire new skills by imitating expert demonstrations, allowing users to communicate their instructions in an intuitive manner. Recent progress in LfD often relies on kinesthetic teaching or teleoperation as the medium for users to specify the demonstrations. Kinesthetic teaching requires physical handling of the robot, while teleoperation demands proficiency with additional hardware. This paper introduces an alternative paradigm for LfD called Diagrammatic Teaching. Diagrammatic Teaching aims to teach robots novel skills by prompting the user to sketch out demonstration trajectories on 2D images of the scene, these are then synthesised as a generative model of motion trajectories in 3D task space. Additionally, we present the Ray-tracing Probabilistic Trajectory Learning (RPTL) framework for Diagrammatic Teaching. RPTL extracts time-varying probability densities from the 2D sketches, applies ray-tracing to find corresponding regions in 3D Cartesian space, and fits a probabilistic model of motion trajectories to these regions. New motion trajectories, which mimic those sketched by the user, can then be generated from the probabilistic model. We empirically validate our framework both in simulation and on real robots, which include a fixed-base manipulator and a quadruped-mounted manipulator.

Figures (8)

• Figure 1: Diagrammatically teaching a quadruped with a mounted arm to shut a drawer, by sketching robot demonstrations over 2D images. The user sketches trajectories of the desired movement over images of the scene (top left and right). The robot then learns to execute the skill and close the drawer (bottom).
• Figure 2: Overview and components of Ray-tracing Probabilistic Trajectory Learning for Diagrammatic Teaching.
• Figure 3: Example of learning to push a box. Top: sketched demonstrations over two different views. Middle: Robot executing the learned skill. Bottom: Densities in 2D view space with time axis collapsed and trajectories from the model in 3D task space.
• Figure 4: Rays traced from cameras at different poses.
• Figure 5: We use Diagrammatic Teaching to teach the robot to follow "R" characters. Left: The $x,y,z$ positions of sampled trajectories over normalised time $t$. The initial positions of the trajectory samples are enforced at the black marker. Right: Three end-effector trajectories, conditioned to start from the current position.