3D Reconstruction from Sketches

TL;DR

This work tackles 3D reconstruction from sketches by integrating a stitching-based front end with a sketch-to-image translation (CycleGAN) and a depth estimator (MegaDepth) to produce textured 3D surfaces. A dataset of image-sketch pairs derived from the Zurich Building Database enables training of the CycleGAN to convert sketches into realistic images suitable for depth inference. The results show that stitching across real, style-variant drawings is brittle, but the single-sketch pathway yields plausible 3D reconstructions and depth maps for a variety of drawings. The approach offers a path to modeling pre-photography scenes and can be extended with more artist diversity and domain-specific fine-tuning of depth estimators to improve robustness and accuracy.

Abstract

We consider the problem of reconstructing a 3D scene from multiple sketches. We propose a pipeline which involves (1) stitching together multiple sketches through use of correspondence points, (2) converting the stitched sketch into a realistic image using a CycleGAN, and (3) estimating that image's depth-map using a pre-trained convolutional neural network based architecture called MegaDepth. Our contribution includes constructing a dataset of image-sketch pairs, the images for which are from the Zurich Building Database, and sketches have been generated by us. We use this dataset to train a CycleGAN for our pipeline's second step. We end up with a stitching process that does not generalize well to real drawings, but the rest of the pipeline that creates a 3D reconstruction from a single sketch performs quite well on a wide variety of drawings.

Figures (8)

• Figure 1: All steps of our pipeline for different drawings. Only sketch (d) required the stitching process.
• Figure 2: Two neighbouring viewpoints of Field Lane at close time periods. The building highlighted in red was demolished between the two drawings. Note the same building highlighted in blue and the timbers highlighted in green that appear in the two drawings. Finding a way to stitch parts of these drawings together would allow us to have the refined details of the drawing on the right with the inclusion of the building that was destroyed from the drawing on the left.
• Figure 3: Our entire pipeline takes in multiple sketches of the same scene with correspondences and outputs a 3D reconstruction of the scene.
• Figure 4: Process to stitch multiple drawings of the same scene together.
• Figure 5: Steps involved in creation of our dataset of sketches of buildings from images in the Zurich Building Image Database.