Skill Generalization with Verbs

TL;DR

This work proposes a method for generalizing manipulation skills to novel objects using verbs that learns a probabilistic classifier that determines whether a given object trajectory can be described by a specific verb.

Abstract

It is imperative that robots can understand natural language commands issued by humans. Such commands typically contain verbs that signify what action should be performed on a given object and that are applicable to many objects. We propose a method for generalizing manipulation skills to novel objects using verbs. Our method learns a probabilistic classifier that determines whether a given object trajectory can be described by a specific verb. We show that this classifier accurately generalizes to novel object categories with an average accuracy of 76.69% across 13 object categories and 14 verbs. We then perform policy search over the object kinematics to find an object trajectory that maximizes classifier prediction for a given verb. Our method allows a robot to generate a trajectory for a novel object based on a verb, which can then be used as input to a motion planner. We show that our model can generate trajectories that are usable for executing five verb commands applied to novel instances of two different object categories on a real robot.

Figures (6)

• Figure 1: Given an observation of an object (before the verb is applied) and a desired verb command, our model generates an object trajectory.
• Figure 2: Diagram of the planning portion of the proposed model. Parameters of the initiation image are extracted and then manipulated by the optimizer, which relies on the categorical cross-entropy loss calculated on the probabilities from the classifier and the target array. Trajectory timestep images (excluding the initiation image) are rendered and given to the classifier, along with the initiation image for computing the loss.
• Figure 3: Overall Verb Accuracy Across Object Categories. We perform $k-$fold cross validation, where there will be $k-1$ object categories used for training the classifier, and a unseen $k$-th object category reserved for testing. Each of the 13 categories take their turn being the $k$-th category. "None" verb trajectories are included in training and testing.
• Figure 4: Average Accuracies for Similar versus Farther Categories. "None" verb trajectories are included in training and testing. For testing the "similar" categories: average of $k$-fold cross validation across categories of Dishwasher, Door, Microwave, Refrigerator, Safe, Washing Machine, where the $k$-th category for testing was one of those, and $k-1$ were training categories. The "farther" categories: average of $k$-fold cross validation when the $k$-th category for testing is chosen out of Box, Laptop, Stapler, Toilet, and Trash Can, and the training categories were Dishwasher, Door, Microwave, Refrigerator, Safe, Washing Machine.
• Figure 5: Examples of trajectory optimizer results. (a) The trajectory optimizer correctly decided to manipulate the joint limit parameter by 0.40 radians for each timestep, thus producing a correct Open. (b) The trajectory optimizer correctly decided to manipulate the $y$ parameter by $-0.10$ for each timestep, thus producing a correct TranslateRight.