Robot Instance Segmentation with Few Annotations for Grasping

TL;DR

This work addresses the challenge of instantiating accurate object masks in cluttered robotic scenes with limited annotations by unifying semi-supervised learning with learning through interaction. It introduces RISE, a model-agnostic framework that uses a memory bank of known instances, pseudo-sequence generation from still images, and a cascade of self-supervised and association losses to learn robust instance representations and segmentation. Key contributions include Mask-to-Box coupling, Multi-Label Matching, and an Optimally Transport-based assignment strategy, resulting in state-of-the-art AP at $AP_{50}$ on ARMBench ($86.37$) and strong performance on OCID with minimal supervision. The approach offers practical impact for lifelong robot perception, enabling grasping and manipulation in changing environments without requiring curated before–after interaction datasets, though it notes limitations when objects are scarcely represented in labeled data.

Abstract

The ability of robots to manipulate objects relies heavily on their aptitude for visual perception. In domains characterized by cluttered scenes and high object variability, most methods call for vast labeled datasets, laboriously hand-annotated, with the aim of training capable models. Once deployed, the challenge of generalizing to unfamiliar objects implies that the model must evolve alongside its domain. To address this, we propose a novel framework that combines Semi-Supervised Learning (SSL) with Learning Through Interaction (LTI), allowing a model to learn by observing scene alterations and leverage visual consistency despite temporal gaps without requiring curated data of interaction sequences. As a result, our approach exploits partially annotated data through self-supervision and incorporates temporal context using pseudo-sequences generated from unlabeled still images. We validate our method on two common benchmarks, ARMBench mix-object-tote and OCID, where it achieves state-of-the-art performance. Notably, on ARMBench, we attain an $\text{AP}_{50}$ of $86.37$, almost a $20\%$ improvement over existing work, and obtain remarkable results in scenarios with extremely low annotation, achieving an $\text{AP}_{50}$ score of $84.89$ with just $1 \%$ of annotated data compared to $72$ presented in ARMBench on the fully annotated counterpart.

Figures (8)

• Figure 1: Pseudo-sequence generation from a single unlabeled image. The input is weakly augmented to produce the "before" and "after" images $x_1$ and $x_2$. To emulate scene interaction, objects are drawn from the object memory bank, transformed, and inserted into the "after" frame. The segmentation model's task is to simultaneously associate objects that persist between frames (subject to occlusion), maintain consistency of object instance embedding, and correctly predict the ground-truth mask of the added objects.
• Figure 2: RISE framework (from left to right). Given an unlabeled image $x$ and a bank of known instances, we perform (1) temporal (blue) and (2) spatial (purple) augmentations. Temporal augmentation adds weak augmentation and inserts $K$ instances from the bank to create $x_1$ (the "before" frame). Another round of weak augmentations, combined with adding/moving/removing a subset of the $K$ added instances, produces $x_2$ (the "after" frame). Spatial augmentations adds strong augmentations to create $x_3$. The three images are batched and fed into the model, where the backbone extracts features that are then encoded into instance embedding. The instance embedding from $x_1$ and $x_2$ are used to compute $\mathcal{L}_{embed}$\ref{['eq:loss_embed']}. The embedding from $x_1$ serve as pseudo-labels against the embedding from $x_3$ in the self-supervised loss $\mathcal{L}_\mathrm{u}$\ref{['eq:unsupervised_loss']}.
• Figure 3: Qualitative results of RISE using ResNet-101 backbone, trained on $1\%$ of the labeled data ($99\%$ treated as unlabeled). Comparing the ground truth (center column) and the predicted masks (right-most column), we see that the majority of large items are accurately segmented, while some of the smaller or heavily occluded objects are occasionally missed. The segmentation masks are continuous, indicating high confidence for every object. Mask boundaries are the only regions with instance--background ambiguity (evident by mild noise at object boundaries).
• Figure 4: Label and Mask Accuracy of pseudo-labels. For both he $x$-axis measures training steps in multiples of $\times 1000$.
• Figure 5: Two dimensional independent $Beta(\alpha=0.5, \beta=0.5)$ distribution representing the spread of instance-bank objects inserted into unlabeled images. The distribution favors placing inserted objects at the periphery of the image, since most images contain most of their information about their center (bright regions denote low probability).