SAda-Net: A Self-Supervised Adaptive Stereo Estimation CNN For Remote Sensing Image Data

TL;DR

This work proposes a self-supervised CNN with self-improving adaptive abilities, which uses the sum of inconsistent points in order to track the network convergence.

Abstract

Stereo estimation has made many advancements in recent years with the introduction of deep-learning. However the traditional supervised approach to deep-learning requires the creation of accurate and plentiful ground-truth data, which is expensive to create and not available in many situations. This is especially true for remote sensing applications, where there is an excess of available data without proper ground truth. To tackle this problem, we propose a self-supervised CNN with self-improving adaptive abilities. In the first iteration, the created disparity map is inaccurate and noisy. Leveraging the left-right consistency check, we get a sparse but more accurate disparity map which is used as an initial pseudo ground-truth. This pseudo ground-truth is then adapted and updated after every epoch in the training step of the network. We use the sum of inconsistent points in order to track the network convergence. The code for our method is publicly available at: https://github.com/thedodo/SAda-Net}{https://github.com/thedodo/SAda-Net

Figures (10)

• Figure 1: Overview of our method. The input consists of two stereo rectified satellite image tiles. Our method does not need any additional input in order to be trained. We use the satellite stereo pipeline software (s2p) in order to project our depth estimation into world coordinates and create a point cloud as well as a digital surface model (dsm).
• Figure 2: This image shows one example of the 2019 IEEE Data Fusion Contest (DFC2019) data_fusion data set where the scene has changed between capturing the ground truth data and the image data. F.l.t.r.: rectified reference image, rectified second image and ground truth disparity map. The large building visible in the panchromatic images is missing in the disparity map.
• Figure 3: First row: AOI from Jacksonville, Florida USA using the RGB bands of the multispectral image instead of the panchromatic image for the sake of better visualization. This image was captured using the WorldView-3 wv3 satellite in january 2015. Second row f.l.t.r: one reference tile of the area of interest, the predicted disparity map and the projected digital surface model that resulted from it.
• Figure 4: Schematic of our training loop.
• Figure 5: Detailed illustration of the CNN and patch creation part of our training loop.