3D Semantic MapNet: Building Maps for Multi-Object Re-Identification in 3D

TL;DR

3D Semantic MapNet is presented - a two-stage re-identification model consisting of a 3D object detector that operates on RGB-D videos with known pose, and a differentiable object matching module that solves correspondence estimation between two sets of 3D bounding boxes.

Abstract

We study the task of 3D multi-object re-identification from embodied tours. Specifically, an agent is given two tours of an environment (e.g. an apartment) under two different layouts (e.g. arrangements of furniture). Its task is to detect and re-identify objects in 3D - e.g. a "sofa" moved from location A to B, a new "chair" in the second layout at location C, or a "lamp" from location D in the first layout missing in the second. To support this task, we create an automated infrastructure to generate paired egocentric tours of initial/modified layouts in the Habitat simulator using Matterport3D scenes, YCB and Google-scanned objects. We present 3D Semantic MapNet (3D-SMNet) - a two-stage re-identification model consisting of (1) a 3D object detector that operates on RGB-D videos with known pose, and (2) a differentiable object matching module that solves correspondence estimation between two sets of 3D bounding boxes. Overall, 3D-SMNet builds object-based maps of each layout and then uses a differentiable matcher to re-identify objects across the tours. After training 3D-SMNet on our generated episodes, we demonstrate zero-shot transfer to real-world rearrangement scenarios by instantiating our task in Replica, Active Vision, and RIO environments depicting rearrangements. On all datasets, we find 3D-SMNet outperforms competitive baselines. Further, we show jointly training on real and generated episodes can lead to significant improvements over training on real data alone.

Figures (15)

• Figure 1: 3D Multi-Object Re-Identification: an agent is provided two tours of an environment (egocentric RGD-D videos with known pose). The two layouts may differ with objects added (red), removed (orange), moved (green) or unchanged (blue). The goal for the agent is to detect and re-identify objects in 3D.
• Figure 2: 3D-SMNet consists of a 3D object detector and a matching module. The 3D object detector qi2019deep takes as input a textured point-cloud representation of the scene and outputs a set of 3D detections along with feature descriptors. The matching module computes similarity scores from the pairwise descriptors and then extends the score matrix with dustbin vectors estimated from an attention mechanism over the two sets of features to capture added/removed objects. The Sinkhorn algorithm sinkhorn1967concerning is then applied to solve the partial assignment problem.
• Figure 3: 3D-SMNet qualitative results on MP3D scenes chang2017matterport3d with inserted YCB calli2015ycb and Google-scanned google_scans assets. 3D-SMNet is able to match 'unchanged' objects like the red examples of the first and second columns. 3D-SMNet is also capable to re-identify objects located at very different locations like the green example of the left column, the green example of the middle column, and the orange example of the right column.
• Figure 4: 3D-SMNet qualitative results on the Replica scenes replica19arxiv. On this zero-shot experiment, 3D-SMNet is able to detect and re-identify 'unchanged' objects (like the plant in purple) and 'moved' objects (like the couch in blue and chair in orange).
• Figure 5: 3D-SMNet qualitative results on the Active Vision dataset replica19arxiv. On this zero-shot experiment on real data, 3D-SMNet is able to detect and re-identify 'unchanged' objects (like the couch in blue) and 'moved' objects (like the two chairs in red and chair in orange).