Towards 3D VR-Sketch to 3D Shape Retrieval

TL;DR

This work reframes 3D shape retrieval around 3D VR-sketch input, proposing a VR-based pipeline to retrieve 3D models from air-drawn sketches. It introduces a dedicated VR data collection, a synthetic 3D sketch generator with controllable abstraction levels, and a comprehensive comparison of multi-view and point-based representations, finding that point-based methods are more robust to sparse, abstract sketches. A reconstruction-based regularization path further improves retrieval by aligning sketch embeddings with the abstraction level of the sketches. Overall, the paper demonstrates that 3D VR-sketch retrieval is viable, with synthetic data generalizing to human sketches and a strong case for using volumetric/point-based representations in this domain.

Abstract

Growing free online 3D shapes collections dictated research on 3D retrieval. Active debate has however been had on (i) what the best input modality is to trigger retrieval, and (ii) the ultimate usage scenario for such retrieval. In this paper, we offer a different perspective towards answering these questions -- we study the use of 3D sketches as an input modality and advocate a VR-scenario where retrieval is conducted. Thus, the ultimate vision is that users can freely retrieve a 3D model by air-doodling in a VR environment. As a first stab at this new 3D VR-sketch to 3D shape retrieval problem, we make four contributions. First, we code a VR utility to collect 3D VR-sketches and conduct retrieval. Second, we collect the first set of $167$ 3D VR-sketches on two shape categories from ModelNet. Third, we propose a novel approach to generate a synthetic dataset of human-like 3D sketches of different abstract levels to train deep networks. At last, we compare the common multi-view and volumetric approaches: We show that, in contrast to 3D shape to 3D shape retrieval, volumetric point-based approaches exhibit superior performance on 3D sketch to 3D shape retrieval due to the sparse and abstract nature of 3D VR-sketches. We believe these contributions will collectively serve as enablers for future attempts at this problem. The VR interface, code and datasets are available at https://tinyurl.com/3DSketch3DV.

Figures (12)

• Figure 1: Our collected VR sketches exhibit variability of sketching styles and levels of details.
• Figure 2: Our VR sketching environment allows to load the reference 3D model, and the user is asked to sketch around it. At any moment the user can hide the 3D model.
• Figure 3: For a subset of shapes from the two categories from ModelNet10: chairs and bathtubs (a), we collect human novices sketches (b). We generate a synthetic dataset of 3D sketches for the 10 shape categories from ModelNet10. We first generate detailed curve networks with FlowRep gori2017flowrep (c) and then apply a set of detail and stroke filtering steps to mimic the appearance of human sketches, where the appearance is controlled by an abstractness parameter $l_a$ (d), defined in Section \ref{['sec:abstraction']}.
• Figure 4: We found little effect of the descriptiveness threshold parameter $d_{max}$ in FlowRep gori2017flowrep on most of the shapes from ModelNet10 dataset.
• Figure 5: The two diagrams on the left demonstrate the heterogeneous and siamese network architectures with NGVNN as a backbone. The two diagrams on the right show the networks based on PointNet++, when trained as described in Section \ref{['sec:metric_learning']} and in Section \ref{['sec:regularization_branch']}.