Single-Shot Metric Depth from Focused Plenoptic Cameras

TL;DR

This work tackles the challenge of recovering metric depth from a single plenoptic camera shot by introducing a two-stage pipeline: first, a Microlens Depth Network predicts sparse metric depth from 23×23 flower-stack microlens patches; then a dense monocular model (Depth Anything) is scale-aligned to produce dense metric depth using Theil–Sen regression to estimate a linear mapping $y = m x + b$. The authors curate the Light Field & Stereo (LFS) Dataset to provide real-world plenoptic images with stereo depth labels for training and evaluation. Experimental results show significant improvements over baselines, including reduced RMSE for sparse depth and superior metric-depth recovery after scale alignment, establishing the feasibility of end-to-end dense metric depth from a single plenoptic shot. The work advances plenoptic-depth estimation by combining specialized learning for microlens-scale depth with robust scale recovery and data-driven densification, enabling practical single-shot depth sensing for robotics and related fields.

Abstract

Metric depth estimation from visual sensors is crucial for robots to perceive, navigate, and interact with their environment. Traditional range imaging setups, such as stereo or structured light cameras, face hassles including calibration, occlusions, and hardware demands, with accuracy limited by the baseline between cameras. Single- and multi-view monocular depth offers a more compact alternative, but is constrained by the unobservability of the metric scale. Light field imaging provides a promising solution for estimating metric depth by using a unique lens configuration through a single device. However, its application to single-view dense metric depth is under-addressed mainly due to the technology's high cost, the lack of public benchmarks, and proprietary geometrical models and software. Our work explores the potential of focused plenoptic cameras for dense metric depth. We propose a novel pipeline that predicts metric depth from a single plenoptic camera shot by first generating a sparse metric point cloud using machine learning, which is then used to scale and align a dense relative depth map regressed by a foundation depth model, resulting in dense metric depth. To validate it, we curated the Light Field & Stereo Image Dataset (LFS) of real-world light field images with stereo depth labels, filling a current gap in existing resources. Experimental results show that our pipeline produces accurate metric depth predictions, laying a solid groundwork for future research in this field.

Figures (6)

• Figure 1: Left: Plenoptic image from a light field camera, displaying the microlens pattern (see detail in Fig. \ref{['fig:flower_stack']}). Center: Corresponding natural image, synthesized from the central viewpoint of the plenoptic camera. Right: Our single-shot metric depth map, at the true scale of the scene.
• Figure 2: Flower stack illustration. Each flower stack is constructed by piling a central microlens and another six in the ring surrounding it. Each stacked microlens is debayered into a 3-channel RGB image.
• Figure 3: Visual results produced through the stereo processing pipeline to obtain suitable ground-truth depth values from stereo images, reprojected onto the plenoptic camera's calibrated reference frame.
• Figure 4: Overview. The Image Processing Toolkit pre-processes captured data, which is then used to train the Microlens Depth Network for sparse metric depth estimation. These are used to refine the inference by Depth Anything, producing a dense metric depth map. Filtered stereo depth serves as the ground truth, following densification and scale alignment.
• Figure 5: Qualitative sparse depth results. Top row: Plenoptic images. Center row: Ground-truth depth from stereo. Bottom row: Depth from our Microlens Depth Network. Note how our depth values are very close to the ground truth.