From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy

TL;DR

XLFM-Former tackles learning-based XLFM reconstruction by uniting physics-informed differentiable rendering with angularly aware self-supervision. It introduces the XLFM-Zebrafish benchmark, the MVM-LF pretraining task, and the ORC Loss to enforce PSF-consistent reconstructions, achieving state-of-the-art PSNR/SSIM on the XLFM-Zebrafish dataset (PSNR 54.04 dB, SSIM 0.9944). The approach demonstrates strong data efficiency, cross-domain generalization (H2B-Nemos), and real-time inference potential, while remaining robust to mild PSF mis-calibration. The work highlights a scalable path for physics-aligned, data-efficient learning in scientific imaging, with potential impact on large-scale neuroscience imaging and beyond.

Abstract

Light field microscopy (LFM) has become an emerging tool in neuroscience for large-scale neural imaging in vivo, notable for its single-exposure volumetric imaging, broad field of view, and high temporal resolution. However, learning-based 3D reconstruction in XLFM remains underdeveloped due to two core challenges: the absence of standardized datasets and the lack of methods that can efficiently model its angular-spatial structure while remaining physically grounded. We address these challenges by introducing three key contributions. First, we construct the XLFM-Zebrafish benchmark, a large-scale dataset and evaluation suite for XLFM reconstruction. Second, we propose Masked View Modeling for Light Fields (MVN-LF), a self-supervised task that learns angular priors by predicting occluded views, improving data efficiency. Third, we formulate the Optical Rendering Consistency Loss (ORC Loss), a differentiable rendering constraint that enforces alignment between predicted volumes and their PSF-based forward projections. On the XLFM-Zebrafish benchmark, our method improves PSNR by 7.7% over state-of-the-art baselines.

Figures (8)

• Figure 1: Our pretraining pipeline for XLFM. The raw light field acquired from the microscope is separated into 27 distinct viewpoints based on physical coordinates. With a 70% probability, we randomly mask a subset of these viewpoints and task the model with reconstructing them. The training is supervised by an $\ell_2$ loss comparing the predicted and ground-truth views.
• Figure 2: The multi-view images used for pretraining an encoder model. For each triplet, we show the masked image (left), our MVM-LF regenerated image (middle), and the ground-truth (right). The masked regions are generated by applying the binary mask complement to the original image.
• Figure 3: Regeneration of XLFM light field images via MVM-LF. The model can still accurately predict the view under appropriate occlusion, indicating that it has learned the global view relationship. Excessive occlusion (90%) causes prediction to crash, indicating that MVM-LF requires a reasonable occlusion ratio to balance information loss and network learning ability.
• Figure 4: Overview of the Swin-XLFM architecture.
• Figure 5: Comparison with state-of-the-art architectures on the XLFM-Zebrafish Dataset. For visualization of Zebrafish sample #1, the PSNR/SSIM values are shown in the top-left corner of each image. Additional examples on samples #2–#6 are provided in the supplementary (Figure \ref{['fig:supp_zebrafish_all']}).