LookUp3D: Data-Driven 3D Scanning

TL;DR

LookUp3D tackles the challenge of fast, high-resolution 3D scanning during dynamic interactions by replacing projector-camera triangulation with a data-driven per-pixel depth-to-color lookup table (LUT). The method calibrates a LUT by sweeping a linear stage over depth and embedding projector imperfections into the dictionary, enabling depth retrieval from a single RGB pattern plus a white flash without explicit projector calibration. The approach achieves up to 450 fps at 1 MP and 1,450 fps at 0.4 MP in controlled lighting, demonstrates accurate reconstruction of high-speed deformations, and enables estimation of physical properties such as gravity and restitution from dynamic sequences. Key contributions include a complete LUT-based SL pipeline (calibration and reconstruction), normalization and denoising strategies to manage noise and storage, residual-based confidence measures, and a detailed hardware prototype that decouples performance from projector optics. This work has practical impact for robotics, computer vision, and physical simulation by providing robust, high-speed 3D data with a simple, reproducible setup that can infer physical quantities from motion-rich scenes.

Abstract

High speed, high-resolution, and accurate 3D scanning would open doors to many new applications in graphics, robotics, science, and medicine by enabling the accurate scanning of deformable objects during interactions. Past attempts to use structured light, time-of-flight, and stereo in high-speed settings have usually required tradeoffs in resolution or inaccuracy. In this paper, we introduce a method that enables, for the first time, 3D scanning at 450 frames per second at 1~Megapixel, or 1,450 frames per second at 0.4~Megapixel in an environment with controlled lighting. The key idea is to use a per-pixel lookup table that maps colors to depths, which is built using a linear stage. Imperfections, such as lens-distortion and sensor defects are baked into the calibration. We describe our method and test it on a novel hardware prototype. We compare the system with both ground-truth geometry as well as commercially available dynamic sensors like the Microsoft Kinect and Intel Realsense. Our results show the system acquiring geometry of objects undergoing high-speed deformations and oscillations and demonstrate the ability to recover physical properties from the reconstructions.

Figures (30)

• Figure 1: Method Overview. Our pipeline consists of two stages: LookUp3D Calibration and LookUp3D Reconstruction. During calibration, we sweep a planar target with a linear stage in fine depth increments. At each step, we use normalized pattern image against a white flash to produce per-pixel RGB observations. These are assembled into a per-pixel lookup table (LUT) that encodes the color-to-depth mapping. Once calibrated, we scan unknown geometry by capturing the same pattern and white flash, normalizing them, and querying the LUT to assign depths via nearest-neighbor search in Euclidean distance.
• Figure 2: Static Reconstructions with Different Hardware. We reconstruct a pawn and a bunny using a high-end DLP projector and an inexpensive LCD projector, both paired with the same camera. Traditional SL reconstruction with 44 Gray Codes deteriorates significantly with the low-quality projector due a reliance on accurate triangulation, whereas LookUp3D with 11 Gray Code channels remains robust.
• Figure 3: Mosaic of Static Scenes. These scenes were captured with our DLP Projector and Atlas Camera in a static setting with controlled lighting to showcase the quality of reconstructions we can achieve with LookUp3D. We show four scenes reconstructed with 9 channels: top left is a color calibration board; top right is a precisely-machined irregular structure; bottom left is a mug; bottom right is a replica of a fish fossil. Each reconstructed cloud is displayed with two zoom-ins of different regions to highlight different shapes, textures, and albedo.
• Figure 4: Mosaic of Static Scenes with Ceiling Lights On. These static scenes were captured with our DLP Projector with ceiling lights on. Even though the lookup table was calibrated in a darkroom, LookUp3D is able to accurately reconstruct the objects. We show a chess board, a paint brush, a 3D-printed house, and a monkey statue, all reconstructed with 11 channels.
• Figure 5: Adapting Reconstruction from Static to Dynamic Setting. We illustrate key algorithmic improvements that enable reconstruction in the high-speed setting, demonstrated on a water balloon impact sequence. (a) The static reconstruction pipeline fails to recover many points with residual below $0.2$. (b) LUT denoising via low-rank approximation improves coverage but remains incomplete. (c) Coarse-to-fine (C2F) search helps recover locally consistent depths, further improving the completeness. (d) Adding temporal consistency (TC) leverages frame-to-frame coherence and helps reconstructing most of the object under challenging deformations.