Learning Generalizable Tool-use Skills through Trajectory Generation

TL;DR

This work proposes to learn a generative model of the tool-use trajectories as a sequence of tool point clouds, which generalizes to different tool shapes and generalizes to various novel tools, significantly outperforming baselines.

Abstract

Autonomous systems that efficiently utilize tools can assist humans in completing many common tasks such as cooking and cleaning. However, current systems fall short of matching human-level of intelligence in terms of adapting to novel tools. Prior works based on affordance often make strong assumptions about the environments and cannot scale to more complex, contact-rich tasks. In this work, we tackle this challenge and explore how agents can learn to use previously unseen tools to manipulate deformable objects. We propose to learn a generative model of the tool-use trajectories as a sequence of tool point clouds, which generalizes to different tool shapes. Given any novel tool, we first generate a tool-use trajectory and then optimize the sequence of tool poses to align with the generated trajectory. We train a single model on four different challenging deformable object manipulation tasks, using demonstration data from only one tool per task. The model generalizes to various novel tools, significantly outperforming baselines. We further test our trained policy in the real world with unseen tools, where it achieves the performance comparable to human. Additional materials can be found on our project website: https://sites.google.com/view/toolgen.

Figures (8)

• Figure 1: Our method ToolGen can solve deformable object manipulation with diverse tasks and goals. It does so by first generating a point cloud trajectory of the desired tool and then aligning the actual tool to the generated point clouds for execution. We train a single model for four different challenging deformable object manipulation tasks. Our model is trained with demonstration data from just a single tool for each task and is able to generalize to various unseen tools.
• Figure 2: Overview of our method: (a) Given an initial observation of the scene $P^o$, the goal $P^g$, and a tool $P^{tool}$, we first leverage the trajectory generation module $G_{traj}$ to generate an ideal tool trajectory accomplishing the task $P^{gen}_{0:H}$. It encompasses two submodules: Initial point cloud generator $G_{reset}$ generating reset pose $P^{gen}_0$ of reconstructed tool and Path generator $G_{path}$ generating $P_{1:H}^{gen}$ (b) We then align the existing tool with the reconstructed tool via sequential pose optimization to extract the pose of the existing tool $T^{opt}_{0:H}$, and we subsequently use inverse kinematics to obtain the actions for the agent to execute.
• Figure 3: We consider 4 tasks: Roll, Cut, Small scoop, and Large scoop. On the left side of each task, we illustrate how the training tool is used to achieve the goal, overlaying the goal on the initial observation. On the right side, we visualize the initial configurations of the training tool and test tools for each task, highlighting the ability of our method to generalize to novel tools.
• Figure 4: Fig. \ref{['fig:results']}: Performance of all the methods across 3 settings. Fig. \ref{['fig:genresults']}: Examples of generated tool trajectories and test tool alignments.
• Figure 5: Example rollouts of ToolGen (ours) compared to the baseline TFN-Traj. The goal configuration of each task is shown on the top right. ToolGen can effectively use the new tool while the baseline struggles.