Coded-E2LF: Coded Aperture Light Field Imaging from Events

TL;DR

To the best of the knowledge, this work is the first to demonstrate that a 4-D light field with pixel-level accuracy can be reconstructed from events alone, and introduces several advancements from the previous work that enable to theoretically support and practically improve light field reconstruction from events alone.

Abstract

We propose Coded-E2LF (coded event to light field), a computational imaging method for acquiring a 4-D light field using a coded aperture and a stationary event-only camera. In a previous work, an imaging system similar to ours was adopted, but both events and intensity images were captured and used for light field reconstruction. In contrast, our method is purely event-based, which relaxes restrictions for hardware implementation. We also introduce several advancements from the previous work that enable us to theoretically support and practically improve light field reconstruction from events alone. In particular, we clarify the key role of a black pattern in aperture coding patterns. We finally implemented our method on real imaging hardware to demonstrate its effectiveness in capturing real 3-D scenes. To the best of our knowledge, we are the first to demonstrate that a 4-D light field with pixel-level accuracy can be reconstructed from events alone. Our software and supplementary video are available from our project website.

Figures (9)

• Figure 1: Overview. Sequence of coding patterns is applied to aperture plane to generate events. Events are accumulated into event images and then used to reconstruct light field through learned reconstruction algorithm (RecNet). Coding patterns are optimized with RecNet.
• Figure 2: Reference-aware event generation: Eq. (9) uses $I^{\mathrm{ref}}_{x,y}$ to better simulate event generation process than Eq. (8).
• Figure 3: Learned coding patterns ($N=4$) with event counts in $E^{(1,2)}$, $E^{(2,3)}$, and $E^{(3,4)}$ and their total. Event counts are per pixel, averaged in BasicLFSR test dataset. Many events are caused before and after black pattern (numbers in bold).
• Figure 4: Visual result with Lego Knights light field: reconstructed top-left views with epipolar plane images along blue/green lines and insets showing view-by-view reconstruction quality in terms of PSNR.
• Figure 5: Scan order for CA+E2VID: blue/red path for first/second scan.