L2D2: Robot Learning from 2D Drawings

TL;DR

This work proposes L2D2, a sketching interface and imitation learning algorithm where humans can provide demonstrations by drawing the task, and finds that L2D2 learns more performant robot policies, requires a smaller dataset, and can generalize to longer-horizon tasks.

Abstract

Robots should learn new tasks from humans. But how do humans convey what they want the robot to do? Existing methods largely rely on humans physically guiding the robot arm throughout their intended task. Unfortunately -- as we scale up the amount of data -- physical guidance becomes prohibitively burdensome. Not only do humans need to operate robot hardware but also modify the environment (e.g., moving and resetting objects) to provide multiple task examples. In this work we propose L2D2, a sketching interface and imitation learning algorithm where humans can provide demonstrations by drawing the task. L2D2 starts with a single image of the robot arm and its workspace. Using a tablet, users draw and label trajectories on this image to illustrate how the robot should act. To collect new and diverse demonstrations, we no longer need the human to physically reset the workspace; instead, L2D2 leverages vision-language segmentation to autonomously vary object locations and generate synthetic images for the human to draw upon. We recognize that drawing trajectories is not as information-rich as physically demonstrating the task. Drawings are 2-dimensional and do not capture how the robot's actions affect its environment. To address these fundamental challenges the next stage of L2D2 grounds the human's static, 2D drawings in our dynamic, 3D world by leveraging a small set of physical demonstrations. Our experiments and user study suggest that L2D2 enables humans to provide more demonstrations with less time and effort than traditional approaches, and users prefer drawings over physical manipulation. When compared to other drawing-based approaches, we find that L2D2 learns more performant robot policies, requires a smaller dataset, and can generalize to longer-horizon tasks. See our project website: https://collab.me.vt.edu/L2D2/

Figures (7)

• Figure 1: Human demonstrating a scooping task to the robot using different teaching paradigms. When using traditional methods to teach the robot, the user needs to manually reset the environment by changing the bowl position and provide demonstrations by physically guiding the robot through the task. We propose L2D2, an approach that synthetically generates diverse environment settings and enables the human to demonstrate the task by drawing a trajectory on the artificial images of the environment. If the robot makes a mistake when executing the learned task, the user provides a few physical corrections to fine-tune the robot's learned behavior. Our proposed approach reduces costly physical interactions and enables humans to teach robots efficiently.
• Figure 2: Interface for demonstrating tasks by sketching robot trajectories. We present this interface to users on a touch-screen device. Users begin by drawing a line starting from the end-effector of the robot on the environment image shown on the left. This line represents the trajectory that the robot's end-effector will follow during the task. Users then specify how the end-effector should rotate by first selecting a point on the line and then selecting the orientation at that point using the $rot_x$, $rot_y$, and $rot_z$ sliders. We provide a visualization of the gripper orientation to help users identify their desired angles. In the same way, users can specify when the gripper should open or close by selecting a point on their line and then choosing the appropriate button.
• Figure 3: Proposed approach for Learning from 2D Drawings (L2D2). The top row outlines our procedure for collecting diverse sketching data. Our approach takes an initial image of the environment as input and creates multiple synthetic images covering a variety of task configurations. We achieve this by detecting relevant objects mentioned by the user using vision-language models (VLMs) and then randomly repositioning those objects in the scene. Users draw on these images to convey the desired task using our interface in Figure \ref{['fig:interface']}. We then use a task-agnostic mapping to convert the 2D points in each sketch to 3D positions in the real world. This ultimately results in a dataset $\tilde{\mathcal{D}}_{R}$ of state-action pairs $(\tilde{s}, \tilde{a})$ reconstructed from the drawings $\xi_{P} \in \mathcal{D}_{P}$. The bottom row outlines our training process. We first train a policy on the reconstructed data using behavior cloning and roll out this policy in the environment. If this policy makes any errors, users physically correct the robot's motion. These corrections result in a small dataset $\mathcal{D}_{R}$ of accurate physical demonstrations. We first use this physical data to refine our 2D-to-3D mapping and improve the quality of demonstrations reconstructed from the sketches. Then we leverage both these datasets to fine-tune the robot's policy and ground the robot's actions in the real world. Together, the diverse set of sketches and a few precise physical demonstrations result in an accurate and generalizable robot policy.
• Figure 4: Results for short-horizon tasks with expert data. (Top) The user is trying to teach the robot to push the bowl to the center of the table (blue region) by drawing the trajectory for the task on the image of the environment. (Bottom) The robot is learning to pick up a block from the drawings provided by the user. We report the success rate for both tasks averaged over $10$ independent rollouts with varying object locations. The error bars show the standard error around the mean (SEM), and $*$ denotes statistical significance $(p<0.05)$. For the push task, L2D2 achieves a higher success rate than L2D2-D and S2S, while for the Lift task L2D2 performs similar to Teleop and outperforms all other baselines.
• Figure 5: Objective results for the user study in Section \ref{['ss:e2']}. Participants teach the robot to perform two tasks in the environment: Scooping and Pick and Place. In each task, they provide physical demonstrations through teleoperation (Teleop) and drawings using our proposed approach (L2D2) and two sketching baselines, RT-Traj and S2S. We record the total time spent by the users in providing demonstrations to the robot and the average success rate of the learned policy evaluated over 10 rollouts. The error bars in the plots show the SEM and $*$ signifies that L2D2 had a significantly better performance than the baseline. Across both tasks, users spend significantly less time providing demonstrations using L2D2, and achieved a significantly higher success rate as compared to RT-Traj and S2S.