Filter-Based Reconstruction of Images from Events

TL;DR

FIBAR introduces a lightweight, asynchronous method to reconstruct brightness images from event camera data using per-pixel temporal filtering and a queue-based spatial filter. The approach is timescale-invariant and CPU-friendly, enabling arbitrary read-out times and practical tasks like fiducial-tag detection and camera calibration, albeit with noisier reconstructions and occasional ghost artifacts compared to neural networks. Through experiments on tag detection, calibration, and inter-pixel threshold analyses, the paper demonstrates that FIBAR can be competitive for specific applications while highlighting its speed advantages and current limitations. The work also provides an appendix on relative contrast thresholds to explain observed noise behaviors and justify the design choices.

Abstract

Reconstructing an intensity image from the events of a moving event camera is a challenging task that is typically approached with neural networks deployed on graphics processing units. This paper presents a much simpler, FIlter Based Asynchronous Reconstruction method (FIBAR). First, intensity changes signaled by events are integrated with a temporal digital IIR filter. To reduce reconstruction noise, stale pixels are detected by a novel algorithm that regulates a window of recently updated pixels. Arguing that for a moving camera, the absence of events at a pixel location likely implies a low image gradient, stale pixels are then blurred with a Gaussian filter. In contrast to most existing methods, FIBAR is asynchronous and permits image read-out at an arbitrary time. It runs on a modern laptop CPU at about 42(140) million events/s with (without) spatial filtering enabled. A few simple qualitative experiments are presented that show the difference in image reconstruction between FIBAR and a neural network-based approach (FireNet). FIBAR's reconstruction is noisier than neural network-based methods and suffers from ghost images. However, it is sufficient for certain tasks such as the detection of fiducial markers. Code is available at https://github.com/ros-event-camera/event_image_reconstruction_fibar

Figures (9)

• Figure 1: Reconstructed (scaled) brightness at the center pixel (319, 239) for a periodic signal (see Sec. \ref{['sec:pixel_thresholds']}). $L^{\mathrm{simple}}(t)$ is obtained by a simple sum of the polarities, see Eq. (\ref{['eq:simple_reconstruction']}). Unequal ON/OFF contrast thresholds result in more OFF than ON events and a downward trend of $L^{\mathrm{simple}}(t)$. $L^{\mathrm{calib}}(t)$ has been detrended by adjusting the threshold $C_{\mathrm{OFF}}$, but still, low-frequency oscillations are present. FIBAR filtering with a cutoff period of $T_{\mathrm{cut}}=100$ removes both the trend and the low frequency distortions.
• Figure 2: Bode magnitude plot of the transfer functions in Eq. (\ref{['eq:transfer_functions']}). The cutoff frequency of $H_\alpha$ and $H_\beta$ has been set to coincide at $\omega_{\mathrm{cut}} = 0.02\pi$ which corresponds to $T_{\mathrm{cut}}=100$ events. The band pass center frequency is also close to $\omega_{\mathrm{cut}}$.
• Figure 3: Frame reconstructed with $r^\mathrm{(fill)}=0.5$ and $T_{\mathrm{cut}}=40$. The marked blue rectangle is shown in Fig. \ref{['fig:fill_ratios']}.
• Figure 4: Image of active pixels (IAP) for fill ratios 0.3 (top), 0.5 (middle) and 0.7 (bottom) for the rectangle marked in Fig. \ref{['fig:fill_ratios_frame']}. Color indicates the number of active events at each pixel. The length of the active event queue grows with the fill ratio, resulting in multiple active events per pixel (middle and bottom). Too low a fill ratio (top) leads to premature spatial filtering of pixels that are still near strong image gradients and will generate further events in the near future. If the fill ratio is too high (bottom), the pixel remains marked as active for so long that close-by image features are already generating events again, thereby delaying the spatial filtering unnecessarily.
• Figure 5: Frames reconstructed with FireNet (left column), FIBAR (center column), and FIBAR without spatial filtering (right column). FireNet reconstructions exhibit much lower noise and therefore allow the detection of small tags further away, but occasionally suffer from persistent artifacts that prevent tag detection. Compare the center and right column to see the value of spatial filtering.