Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D Annotation Of Arbitrary Objects In The Wild

Kenneth Blomqvist, Julius Hietala

TL;DR

A data annotation pipeline based on SLAM, 3D reconstruction, and 3D-to-2D geometry is proposed, which allows creating 3D and 2D bounding boxes, along with per-pixel annotations of arbitrary objects without needing accurate 3D models of the objects prior to data collection and annotation.

Abstract

Recent years have produced a variety of learning based methods in the context of computer vision and robotics. Most of the recently proposed methods are based on deep learning, which require very large amounts of data compared to traditional methods. The performance of the deep learning methods are largely dependent on the data distribution they were trained on, and it is important to use data from the robot's actual operating domain during training. Therefore, it is not possible to rely on pre-built, generic datasets when deploying robots in real environments, creating a need for efficient data collection and annotation in the specific operating conditions the robots will operate in. The challenge is then: how do we reduce the cost of obtaining such datasets to a point where we can easily deploy our robots in new conditions, environments and to support new sensors? As an answer to this question, we propose a data annotation pipeline based on SLAM, 3D reconstruction, and 3D-to-2D geometry. The pipeline allows creating 3D and 2D bounding boxes, along with per-pixel annotations of arbitrary objects without needing accurate 3D models of the objects prior to data collection and annotation. Our results showcase almost 90% Intersection-over-Union (IoU) agreement on both semantic segmentation and 2D bounding box detection across a variety of objects and scenes, while speeding up the annotation process by several orders of magnitude compared to traditional manual annotation.

3D Annotation Of Arbitrary Objects In The Wild

TL;DR

A data annotation pipeline based on SLAM, 3D reconstruction, and 3D-to-2D geometry is proposed, which allows creating 3D and 2D bounding boxes, along with per-pixel annotations of arbitrary objects without needing accurate 3D models of the objects prior to data collection and annotation.

Abstract

Recent years have produced a variety of learning based methods in the context of computer vision and robotics. Most of the recently proposed methods are based on deep learning, which require very large amounts of data compared to traditional methods. The performance of the deep learning methods are largely dependent on the data distribution they were trained on, and it is important to use data from the robot's actual operating domain during training. Therefore, it is not possible to rely on pre-built, generic datasets when deploying robots in real environments, creating a need for efficient data collection and annotation in the specific operating conditions the robots will operate in. The challenge is then: how do we reduce the cost of obtaining such datasets to a point where we can easily deploy our robots in new conditions, environments and to support new sensors? As an answer to this question, we propose a data annotation pipeline based on SLAM, 3D reconstruction, and 3D-to-2D geometry. The pipeline allows creating 3D and 2D bounding boxes, along with per-pixel annotations of arbitrary objects without needing accurate 3D models of the objects prior to data collection and annotation. Our results showcase almost 90% Intersection-over-Union (IoU) agreement on both semantic segmentation and 2D bounding box detection across a variety of objects and scenes, while speeding up the annotation process by several orders of magnitude compared to traditional manual annotation.

Paper Structure

This paper contains 16 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. 2D Annotation and Active Learning
  4. 3D Annotation
  5. METHOD
  6. Localization, Mapping and Dense Reconstruction
  7. GUI-based annotation
  8. Label Extraction
  9. EXPERIMENTS
  10. Label Accuracy
  11. Downstream Task
  12. RESULTS
  13. Label quality
  14. Labeling efficiency
  15. Downstream task
  16. ...and 1 more sections

Figures (4)

  • Figure 1: (a) Objects are annotated in a 3d graphical user interface with bounding boxes. (b) Bounding boxes and per-pixel labels are projected back to the original images.
  • Figure 2: All classes in our dataset visualized: Barriers (A), Shoes (B), Pillows (C), Scooters (D), Boxes (E), Food items (F), Poles (G), Bottles (H). The first row in green shows the hand labeled human semantic segmentation mask. The second row shows the scene reconstruction in the graphical user interface. The third row in blue shows segmentation masks, generated by our method.
  • Figure 3: The annotated bounding boxes rendered on top of images sampled from our dataset.
  • Figure 4: Qualitative examples of model predictions (bottom row) compared to ground truth labels (top row)