Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow

Hongyu Wen, Erich Liang, Jia Deng

TL;DR

This paper introduces LayeredFlow, a real world benchmark containing multi-layer ground truth annotation for optical flow of non-Lambertian objects and proposes a new task called multi-layer optical flow, using LayeredFlow as evaluation data.

Abstract

Achieving 3D understanding of non-Lambertian objects is an important task with many useful applications, but most existing algorithms struggle to deal with such objects. One major obstacle towards progress in this field is the lack of holistic non-Lambertian benchmarks -- most benchmarks have low scene and object diversity, and none provide multi-layer 3D annotations for objects occluded by transparent surfaces. In this paper, we introduce LayeredFlow, a real world benchmark containing multi-layer ground truth annotation for optical flow of non-Lambertian objects. Compared to previous benchmarks, our benchmark exhibits greater scene and object diversity, with 150k high quality optical flow and stereo pairs taken over 185 indoor and outdoor scenes and 360 unique objects. Using LayeredFlow as evaluation data, we propose a new task called multi-layer optical flow. To provide training data for this task, we introduce a large-scale densely-annotated synthetic dataset containing 60k images within 30 scenes tailored for non-Lambertian objects. Training on our synthetic dataset enables model to predict multi-layer optical flow, while fine-tuning existing optical flow methods on the dataset notably boosts their performance on non-Lambertian objects without compromising the performance on diffuse objects. Data is available at https://layeredflow.cs.princeton.edu.

LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow

TL;DR

This paper introduces LayeredFlow, a real world benchmark containing multi-layer ground truth annotation for optical flow of non-Lambertian objects and proposes a new task called multi-layer optical flow, using LayeredFlow as evaluation data.

Abstract

Achieving 3D understanding of non-Lambertian objects is an important task with many useful applications, but most existing algorithms struggle to deal with such objects. One major obstacle towards progress in this field is the lack of holistic non-Lambertian benchmarks -- most benchmarks have low scene and object diversity, and none provide multi-layer 3D annotations for objects occluded by transparent surfaces. In this paper, we introduce LayeredFlow, a real world benchmark containing multi-layer ground truth annotation for optical flow of non-Lambertian objects. Compared to previous benchmarks, our benchmark exhibits greater scene and object diversity, with 150k high quality optical flow and stereo pairs taken over 185 indoor and outdoor scenes and 360 unique objects. Using LayeredFlow as evaluation data, we propose a new task called multi-layer optical flow. To provide training data for this task, we introduce a large-scale densely-annotated synthetic dataset containing 60k images within 30 scenes tailored for non-Lambertian objects. Training on our synthetic dataset enables model to predict multi-layer optical flow, while fine-tuning existing optical flow methods on the dataset notably boosts their performance on non-Lambertian objects without compromising the performance on diffuse objects. Data is available at https://layeredflow.cs.princeton.edu.
Paper Structure (42 sections, 4 equations, 6 figures, 10 tables)

This paper contains 42 sections, 4 equations, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Non-Lambertian Benchmarks and Datasets
  4. Synthetic Non-Lambertian Datasets
  5. Real-World Non-Lambertian Benchmarks
  6. See-Through Methods and Benchmarks
  7. Optical Flow Datasets
  8. Synthetic Optical Flow Datasets.
  9. Real-World Optical Flow Benchmarks.
  10. Optical Flow Methods.
  11. Collecting LayeredFlow's Real World Benchmark
  12. Introduction to AprilTag
  13. Data Acquisition and Annotation
  14. Camera Setup and Calibration.
  15. Image Capturing Pipeline.
  16. ...and 27 more sections

Figures (6)

  • Figure 1: Gallery of our non-Lambertian real world benchmark. Our benchmark encompasses 185 indoor and outdoor scenes and 360 different objects with 2000 images. By using a stereo camera and carefully attaching and removing AprilTags, we acquire accurate multi-layer optical flow and stereo measurements.
  • Figure 2: Showcase of our synthetic dataset with ground truth annotations. Left: a sample synthetic image. Right: Multi-layer optical flow and 3D positions in world coordinates.
  • Figure 3: Image Capturing Pipeline. (a) The original stereo image pair. (b) AprilTags are carefully attached to the scene, allowing disparity measurements (yellow arrow). (c) The scene is altered by changing the positions and orientations of scene objects and the stereo camera system. Optical flow is measured via AprilTags (orange arrow). (d) AprilTags are detached, yielding the final tagless stereo image pair.
  • Figure 4: Gallery of our synthetic dataset. Our synthetic dataset is generated from modified versions of 30 high-quality indoor scenes designed by artists. To increase the frequency of non-Lambertian objects and diversity of images, we randomly modify material properties, change scene lighting, and insert additional objects.
  • Figure 5: Synthetic dataset generation pipeline. We perform these steps to boost the frequency of non-Lambertian object appearances and the diversity of rendered images.
  • ...and 1 more figures