Online Object Representations with Contrastive Learning

TL;DR

This work tackles robust, label-free learning of object representations for robots by exploiting self-supervised contrastive learning on monocular video. It introduces the Object-Contrastive Network (OCN), which learns to disentangle object attributes by pulling together embeddings of the same object across frames while pushing apart others, using only object proposals. The method enables online adaptation, demonstrated by dramatic reductions in object identification error during ongoing observation and by a robotic pointing task that generalizes object attributes to unseen objects. The authors validate OCNs on real, automatically gathered, and synthetic data, showing competitive performance to supervised baselines and clear benefits for autonomy and scalability in robotics.

Abstract

We propose a self-supervised approach for learning representations of objects from monocular videos and demonstrate it is particularly useful in situated settings such as robotics. The main contributions of this paper are: 1) a self-supervising objective trained with contrastive learning that can discover and disentangle object attributes from video without using any labels; 2) we leverage object self-supervision for online adaptation: the longer our online model looks at objects in a video, the lower the object identification error, while the offline baseline remains with a large fixed error; 3) to explore the possibilities of a system entirely free of human supervision, we let a robot collect its own data, train on this data with our self-supervise scheme, and then show the robot can point to objects similar to the one presented in front of it, demonstrating generalization of object attributes. An interesting and perhaps surprising finding of this approach is that given a limited set of objects, object correspondences will naturally emerge when using contrastive learning without requiring explicit positive pairs. Videos illustrating online object adaptation and robotic pointing are available at: https://online-objects.github.io/.

Figures (13)

• Figure 1: The longer our model looks at objects in a video, the lower the object identification error. Top: example frames of a work bench video along with the detected objects. Bottom: result of online training on the same video. Our model self-supervises object representations as the video progresses and converges to 2% error while the offline baseline remains at 52% error.
• Figure 2: Object-Contrastive Networks (OCN): by attracting nearest neighbors in embedding space and repulsing others using metric learning, continuous object representations naturally emerge. In a video collected by a robot looking at a table from different viewpoints, objects are extracted from random pairs of frames. Given two lists of objects, each object is attracted to its closest neighbor while being pushed against all other objects. Noisy repulsion may occur when the same object across viewpoint is not matched against itself. However the learning still converges towards disentangled and semantically meaningful object representations.
• Figure 3: Models and baselines: for comparison purposes all models evaluated in Sec. \ref{['sec:results']} share the same architecture of a standard ResNet50 model followed by additional layers. While the architectures are shared, the weights are not across models. While the unsupervised model (left) does not require supervision labels, the 'softmax' baseline as well as the supervised evaluations (right) use attributes labels provided with each object. We evaluate the quality of the embeddings with two types of classifiers: linear and nearest neighbor.
• Figure 4: Six of the environments we used for our self-supervised online experiment. Top: living room, office, kitchen. Bottom: one of our more challenging scenes, and two examples of the Epic-Kitchens Damen2018EPICKITCHENS dataset.
• Figure 6: Comparison of identifying objects with ResNet50 (top) and OCN (bottom) embeddings for the environments kids room (left) and challenging (right). Red bounding boxes indicate a mismatch of the ground truth and associated index.