High-speed Imaging through Turbulence with Event-based Light Fields

Abstract

This work introduces and demonstrates the first system capable of imaging fast-moving extended non-rigid objects through strong atmospheric turbulence at high frame rate. Event cameras are a novel sensing architecture capable of estimating high-speed imagery at thousands of frames per second. However, on their own event cameras are unable to disambiguate scene motion from turbulence. In this work, we overcome this limitation using event-based light field cameras: By simultaneously capturing multiple views of a scene, event-based light field cameras and machine learning-based reconstruction algorithms are able to disambiguate motion-induced dynamics, which produce events that are strongly correlated across views, from turbulence-induced dynamics, which produce events that are weakly correlated across view. Tabletop experiments demonstrate event-based light field can overcome strong turbulence while imaging high-speed objects traveling at up to 16,000 pixels per second.

Figures (15)

• Figure 1: Method overview. We introduce an event-based light field camera to image through turbulence at high speed. The event camera avoids motion blur by detecting brightness changes at microsecond time resolution. The light field offers cross-view constraints on the scene.
• Figure 2: Image model for event-based light field camera under turbulence. A light field camera captures $N$ simultaneous sub-aperture views, each observing the same scene through an independent turbulence realization, which injects different turbulence-induced fluctuations across views while leaving the scene contribution identical.
• Figure 3: Network architecture. Each sub-aperture event stream is converted to a voxel grid $E_k^{(i)} \in \mathbb{R}^{B \times H \times W}$; the $N$ grids are stacked along the channel dimension to form the joint input $\tilde{E}_k \in \mathbb{R}^{NB \times H \times W}$. A recurrent convolutional encoder-decoder processes the stacked input window by window and outputs a reconstructed frame $\hat{V}_k$.
• Figure 4: Simulation pipeline. Turbulence is added to clean frames using a turbulence simulator based on the phase-to-space (P2S) transform turbulence_p2s. V2E converts turbulent frames to events v2e. The process is repeated 9 times to simulate a $3 \times 3$ light field. The network is trained on simulated data and generalizes to tabletop experimental settings.
• Figure 5: Tabletop experimental setup. Real turbulence is generated from temperature fluctuations caused by candles and a space heater. We placed a beam splitter behind the kaleidoscope to simultaneously capture events and reference CMOS frames.