Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols

TL;DR

The Yale-CMU-Berkeley (YCB) Object and Model set is presented, intended to be used to facilitate benchmarking in robotic manipulation, prosthetic design and rehabilitation research, and a comprehensive literature survey on existing benchmarks and object datasets is presented.

Abstract

In this paper we present the Yale-CMU-Berkeley (YCB) Object and Model set, intended to be used to facilitate benchmarking in robotic manipulation, prosthetic design and rehabilitation research. The objects in the set are designed to cover a wide range of aspects of the manipulation problem; it includes objects of daily life with different shapes, sizes, textures, weight and rigidity, as well as some widely used manipulation tests. The associated database provides high-resolution RGBD scans, physical properties, and geometric models of the objects for easy incorporation into manipulation and planning software platforms. In addition to describing the objects and models in the set along with how they were chosen and derived, we provide a framework and a number of example task protocols, laying out how the set can be used to quantitatively evaluate a range of manipulation approaches including planning, learning, mechanical design, control, and many others. A comprehensive literature survey on existing benchmarks and object datasets is also presented and their scope and limitations are discussed. The set will be freely distributed to research groups worldwide at a series of tutorials at robotics conferences, and will be otherwise available at a reasonable purchase cost. It is our hope that the ready availability of this set along with the ground laid in terms of protocol templates will enable the community of manipulation researchers to more easily compare approaches as well as continually evolve benchmarking tests as the field matures.

Figures (17)

• Figure 1: Food items in the YCB Object Set: back row: coffee can, cereal box, cracker box, chips can, box of sugar; middle row: mustard container, chocolate pudding box, tomato soup can, gelatin box, potted meat can; front: plastic fruit (lemon, apple, pear, orange, banana, peach, strawberries, plum).
• Figure 2: Kitchen items in the YCB Object Set: back row: pitcher, plastic wine glass, abrasive sponge, glass cleaner; front: enamelcoated metal dishware (bowl, coffee mug and plate), eating utensils (knife, spoon, and fork), cooking skillet with glass lid, plastic turner.
• Figure 3: Tool items in the YCB Object Set: back: wood blocks (four items, three sizes), power drill; front: nails (three sizes), hammer, plastic bolt and nut; adjustable wrench, wood screw, phillips and flat screwdrivers, padlock and keys, spring clamps (five sizes), scissors, markers (three sizes/types).
• Figure 4: Shape items in the YCB Object Set: back: stacking blocks (set of 10), credit card blank, foam bricks (set of 3), spherical objects (softball, tennis ball, racquetball, golf ball, marble assortment); front: plastic chain, nylon rope, washer assortment (7 sizes), dice (set of 12).
• Figure 5: (left) Box-and-blocks test objects: set of$\mathbf{1 0 0}$ wooden cubes, two containers and height obstacle (container lid) between them. (right) 9-hole peg test: wooden pegs are placed in