Frequency Cam: Imaging Periodic Signals in Real-Time

TL;DR

This work tackles per-pixel frequency detection in event-based cameras by introducing a fully asynchronous approach that reconstructs brightness with a second-order digital IIR filter and then detects zero-level crossings, augmented by a dark-noise filter for robustness. The method shows robust fundamental-frequency estimates up to $64\mathrm{kHz}$ on a single pixel, while full-sensor operation is bandwidth-limited, highlighting the benefit of hardware proximity to the sensor. Frequency Cam, an open-source ROS node, achieves real-time performance and yields frequency images that qualitatively align with Prophesee’s closed-source vibration analysis, offering a practical tool for vibration analysis and frequency visualization in neuromorphic imaging. The study also documents practical challenges such as readout bandwidth constraints and lens flare in high dynamic range scenes, guiding future hardware- and algorithm-level improvements for wide-field frequency imaging.

Abstract

Due to their high temporal resolution and large dynamic range, event cameras are uniquely suited for the analysis of time-periodic signals in an image. In this work we present an efficient and fully asynchronous event camera algorithm for detecting the fundamental frequency at which image pixels flicker. The algorithm employs a second-order digital infinite impulse response (IIR) filter to perform an approximate per-pixel brightness reconstruction and is more robust to high-frequency noise than the baseline method we compare to. We further demonstrate that using the falling edge of the signal leads to more accurate period estimates than the rising edge, and that for certain signals interpolating the zero-level crossings can further increase accuracy. Our experiments find that the outstanding capabilities of the camera in detecting frequencies up to 64kHz for a single pixel do not carry over to full sensor imaging as readout bandwidth limitations become a serious obstacle. This suggests that a hardware implementation closer to the sensor will allow for greatly improved frequency imaging. We discuss the important design parameters for fullsensor frequency imaging and present Frequency Cam, an open-source implementation as a ROS node that can run on a single core of a laptop CPU at more than 50 million events per second. It produces results that are qualitatively very similar to those obtained from the closed source vibration analysis module in Prophesee's Metavision Toolkit. The code for Frequency Cam and a demonstration video can be found at https://github.com/ros-event-camera/frequency_cam

Figures (16)

• Figure 1: Frequency image of a quad rotor in flight, with color coded frequencies (in Hz). The top image was obtained with a filter of $T_{\mathrm{cut}}=5$ whereas the bottom is generated using Prophesee's proprietary vibration analysis module. The image has been centered and cropped to a size of 242 x 281 for ease of viewing. Grey color indicates pixels for which no frequency could be determined, but that have had events during the readout period (10ms). Both images are qualitatively very similar.
• Figure 2: Frequency estimation with the baseline method for a 50Hz square wave signal. The top panel shows the raw events, and the two different ways to estimate the signal period. Note that although the brightness changes abruptly between up and down, the ON and OFF events spread out due to "motion blur", and much more so for ON events (see hu_liu, section 3.1). The bottom panel shows the dramatically lower variance ($\sigma = 1.1\mathrm{us}$, green line) when using the ON/OFF transition for period estimation as opposed to OFF/ON ($\sigma=260\mathrm{us}$, blue line).
• Figure 3: Multiple periods of a single pixel signal with fundamental frequency of 100hz. The top panel shows the raw signal, for which the baseline frquency detection fails. The middle panel displays the (non-stationary) naive illumination reconstruction (cumulative sum of polarities). The bottom panel shows the reconstruction obtained after removing the global (but not local!) trend by adjusting the OFF threshold to $C_{\mathrm{OFF}}=1.88$. Note the DC component during the first cycle and the drift due to the $C_{\mathrm{ON}}$/$C_{\mathrm{OFF}}$ ratio varying with time.
• Figure 4: Bode magnitude plot of the transfer functions of Eq.(\ref{['eq:transfer_functions']}), but normalized to unity at their respective maxima. The coefficients $\alpha=0.51$ and $\beta=0.54$ have been set according to (\ref{['eq:alpha_cut']}), (\ref{['eq:beta_cut']}) to result in a identical cutoff (-3dB) frequency of $\omega_{\mathrm{cut}}=0.2 \pi$ for both high and low pass, resulting in a bandpass center frequency close to $\omega_{\mathrm{cut}}$.
• Figure 5: Single-pixel reconstructed brightness $\tilde{L}(t)$ when the filter (\ref{['eq:filter_recursion_ab']}) is applied with different cutoff periods $T_{\mathrm{cut}}$ to the signal shown in Fig. \ref{['fig:simple_detrend']}. The orange line with $T_{\mathrm{cut}}=144$ corresponds to the recommended setting according to (\ref{['eq:setting_tcut']}). Notice how the DC component is removed faster than for $T_{\mathrm{cut}}=288$ (green line) while still providing a sufficiently high quality reconstruction. The blue line ($T_{\mathrm{cut}}=72$) shows how the reconstruction suffers when setting the cutoff period too small, eliminating low frequencies too aggressively.