Event fields: Capturing light fields at high speed, resolution, and dynamic range

TL;DR

This work develops the underlying mathematical framework for Event Fields and introduces two foundational frameworks to capture them practically: spatial multiplexing to capture temporal derivatives and temporal multiplexing to capture angular derivatives.

Abstract

Event cameras, which feature pixels that independently respond to changes in brightness, are becoming increasingly popular in high-speed applications due to their lower latency, reduced bandwidth requirements, and enhanced dynamic range compared to traditional frame-based cameras. Numerous imaging and vision techniques have leveraged event cameras for high-speed scene understanding by capturing high-framerate, high-dynamic range videos, primarily utilizing the temporal advantages inherent to event cameras. Additionally, imaging and vision techniques have utilized the light field-a complementary dimension to temporal information-for enhanced scene understanding. In this work, we propose "Event Fields", a new approach that utilizes innovative optical designs for event cameras to capture light fields at high speed. We develop the underlying mathematical framework for Event Fields and introduce two foundational frameworks to capture them practically: spatial multiplexing to capture temporal derivatives and temporal multiplexing to capture angular derivatives. To realize these, we design two complementary optical setups one using a kaleidoscope for spatial multiplexing and another using a galvanometer for temporal multiplexing. We evaluate the performance of both designs using a custom-built simulator and real hardware prototypes, showcasing their distinct benefits. Our event fields unlock the full advantages of typical light fields-like post-capture refocusing and depth estimation-now supercharged for high-speed and high-dynamic range scenes. This novel light-sensing paradigm opens doors to new applications in photography, robotics, and AR/VR, and presents fresh challenges in rendering and machine learning.

Figures (14)

• Figure 1: Result of template matching. (a) Two templates are selected from the reconstructed light field image—one in the foreground and one in the background. A template matching algorithm is applied to each template across all light field images to determine the shift amounts in (x,y). (b) The calculated shift amounts are plotted, forming a circular pattern consistent with the scanning curve, validating our concept.
• Figure 2: Blender plugin for physics-based event field rendering.
• Figure 2: Static scene refocusing. Two distinct static scenes are captured using our galvanometer design at a scan speed of 1 Hz. E2VID is used to reconstruct 100 light field frames. For each scene, we present the reconstructed refocusing result on two selected regions.
• Figure 3: Kaleidoscope design. (a) Ray sketch illustrating the light path through the rectangular kaleidoscope system, where rays from different angles are directed towards the event camera. (b) Hardware setup showing the main lens, rectangular kaleidoscope, beam splitter, event camera, and RGB camera. The main lens focuses the scene onto the kaleidoscope, which splits the view into multiple angles. The beam splitter directs part of the light to the RGB camera for standard imaging and part to the event camera to capture high-frequency events with angular differentiation.
• Figure 3: Depth calibration. We calibrate 7 depth points using an LED light placed at distances ranging from 15 inches to 100 inches. The depth (in inches) is linearly fitted to the disparity (in pixels) within this range, serving as our calibrated ratio. For new scenes, we detect disparity using a depth-from-focus algorithm and convert it into real-world depth using this calibrated relationship.