Four-dimensional video imaging via generative deep learning and a diffuser-encoded image sensor

TL;DR

This work uses 4DCam to image a live Betta splendens fish, uncovering polarization-dependent color modulations that remain invisible to conventional cameras and experimentally shows that the 4D information encoded in the scatterograms markedly improves material discrimination.

Abstract

Light carries rich information across space, spectrum, polarization, and time, yet conventional cameras capture only a narrow projection of this multidimensional structure. A thin diffuser encodes wavelength-dependent information into single-shot scatterograms, captured by a polarization-resolving CMOS sensor that simultaneously measures four linear polarization states. We use 4DCam to image a live Betta splendens fish, uncovering polarization-dependent color modulations that remain invisible to conventional cameras. We experimentally show that the 4D information encoded in the scatterograms markedly improves material discrimination, achieving 96% accuracy for textile classification and 90% for camouflage detection, compared with 70% and 80%, respectively, using 3D hyperspectral imaging alone. Built entirely from passive optics, 4DCam seamlessly integrates physical encoding, generative decoding, and direct inference, enabling real-time, information-complete optical sensing.

Figures (6)

• Figure 1: 4DCam for hyperspectral–polarimetric video imaging. (a) A polarization-resolving CMOS sensor is augmented with a thin ground-glass diffuser placed in close proximity to the image plane, which encodes wavelength-dependent scatterogram patterns for each polarization channel. (b) Deep neural networks trained for specific application domains reconstruct four-dimensional (4D) hyperspectral–polarimetric datacubes from single grayscale frames. Example from the Invertebrate-net demonstrates simultaneous recovery of four linear polarization states ($0^\circ$, $45^\circ$, $90^\circ$, $135^\circ$) across 106 spectral channels. (c) Exemplary polarization-averaged sRGB renderings of 4D outputs of separate networks trained on distinct object classes—invertebrates, produce, resolution targets, fossils, and banknotes—as well as a unified model trained across all classes are shown.
• Figure 2: 4DCam video imaging of a live Betta Splendens at 35 frames per second. (a) Overview image of the specimen. Representative frames from the reconstructed 4D hyperspectral–polarimetric video, showing spectral ($\lambda$), and polarization (b) $0^{\circ}$ and (c) $90^{\circ}$channels (see Supplementary Videos 1 and 2). Reconstructions were generated using the unified model, which had not encountered similar dynamic, reflective scenes during training. Although no ground-truth data exist at these frame rates, the predicted voxel-wise uncertainty remains comparable to that observed for the training datasets, indicating high reconstruction fidelity and uncertainty in the recovered spectral–polarimetric dynamics.
• Figure 3: Uncertainty-aware hyperspectral reconstructions. (a) Left to right: Ground-truth (GT), reconstructed, and uncertainty maps at $\lambda=488$ nm for polarization $\theta=0^\circ$. (b) Spectra from labeled points P1–P3 demonstrate close agreement between ground truth (solid lines) and reconstruction (dashed lines), with shaded regions denoting $\pm\sigma$ intervals. (c) Full hyperspectral stack and panchromatic image. (d) Spectra from pixel (85,60) reconstructed by five independently trained probabilistic networks closely match both the ground truth and one another. (e) Laplacian probability density functions (PDFs) at $\lambda=488$ nm quantify ensemble variability, showing narrow, overlapping distributions indicative of calibrated uncertainty and consistent predictive behavior.
• Figure 4: Spectro-polarimetric information enhances effective spatial resolution. (a–c) Illustration showing that two spatially unresolved features, indistinguishable in conventional intensity images, become separable when analysed across additional optical dimensions: (a) through differences in spectral channels, (b) through polarization contrast, or (c) through combined spectro–polarimetric signatures. Harnessing this higher-dimensional information enables effective separation of sub-resolution features. Experimental demonstration using a radial resolution chart. The panchromatic, polarization-averaged baseline image (d) provides limited contrast, whereas the spectro-polarimetric difference image, $|I_{0^{\circ},\lambda=454\,\text{nm}} - I_{90^{\circ},\lambda=738\,\text{nm}}|$, (e) reveals substantially finer structure. Line pairs spaced by 4 sensor pixels (corresponding to an angular resolution of $\approx9$ mrad) exhibit an increase in contrast from 4% to 24%, demonstrating resolution enhancement through spectro-polarimetric information (see Supplement Section 9). Validation on a natural specimen, a butterfly wing, further highlights this effect. The baseline panchromatic polarization-averaged image (f) shows low contrast, while the spectro-polarimetric difference image, $|I_{0^{\circ},\lambda=730\,\text{nm}} - I_{90^{\circ},\lambda=470\,\text{nm}}|$, (g) sharply delineates microstructural boundaries, including a narrow dark line (blue arrow) about 1 sensor pixel wide, revealing features beyond the native spatial resolution of the imager.
• Figure 5: Classification of fabrics using 4DCam data. (a) Average classification accuracy for networks trained on spectral, polarimetric, and fused spectro–polarimetric inputs, as well as on raw scatterograms. All results except the latter are derived from reconstructed hyperspectral–polarimetric images. The uncropped scatterograms were trained on the MobileNetV3 classifier (see Supplement Section 12). (b) Mean reflectance spectra from regions of interest for cotton, felt, and nylon. Dashed lines denote ground truth, solid lines indicate mean reconstructed spectra, and shaded regions represent $\pm1 \sigma$ uncertainty. (c) Representative polarization-resolved spectra from the same ROI example shown in (b), illustrating slight spectral variation across polarization states. Together, these results demonstrate that integrating spectral and polarization cues enhances discriminative power for material classification.