Event Topology-based Visual Microphone for Amplitude and Frequency Reconstruction

TL;DR

This work presents an event topology-based visual microphone that reconstructs vibrations directly from raw event streams without external illumination, and captures the intrinsic structure of event data to recover both amplitude and frequency with high fidelity.

Abstract

Accurate vibration measurement is vital for analyzing dynamic systems across science and engineering, yet noncontact methods often balance precision against practicality. Event cameras offer high-speed, low-light sensing, but existing approaches fail to recover vibration amplitude and frequency with sufficient accuracy. We present an event topology-based visual microphone that reconstructs vibrations directly from raw event streams without external illumination. By integrating the Mapper algorithm from topological data analysis with hierarchical density-based clustering, our framework captures the intrinsic structure of event data to recover both amplitude and frequency with high fidelity. Experiments demonstrate substantial improvements over prior methods and enable simultaneous recovery of multiple sound sources from a single event stream, advancing the frontier of passive, illumination-free vibration sensing.

Figures (5)

• Figure 1: Event camera that records changes in brightness only (red: positive events for brightness increases, blue: negative events for brightness decreases). This function provides a high temporal resolution.
• Figure 2: Overview of the proposed vibration reconstruction pipeline. The process begins with (a) recorded data events for target vibration, where raw events are captured as a point cloud in $x$, $y$, and $t$ space. The red box in (a) indicates the region of interest (ROI) used for vibration reconstruction. In our method, the ROI is centered at the pixel with the highest event density and oriented along the vibration direction. This event data are then (b) filtered along the $y$-axis and split into multiple intervals, creating distinct temporal segments. Subsequently, (c) HDBSCAN clustering is applied within each interval. In (d), the centroid of each cluster is computed, and these points are chronologically connected by the blue line to form a 3D vibration trajectory. Finally, (e) this trajectory is projected onto the y-t plane, discarding horizontal information to produce the final vibration waveform.
• Figure 3: Reconstruction accuracy as a function of ROI dimensions (width, height) for three vibration test conditions: Test Case 1 (increasing amplitude signal), Test Case 2 (decreasing amplitude signal), and Test Case 3 (audio signal). (a-c) Normalized cross-correlation maps and (d-f) MVA error maps for Test Cases 1, 2, and 3, respectively.
• Figure 4: Comparison of vibration waveforms reconstructed through various methods and LDV measurements. The orange traces in all subplots represent the ground truth vibration measured by an LDV. The blue traces show the reconstructed vibration waveforms from different methods: (a) Abe et al.'s method (frame-based approach), (b) Niwa et al.'s method (event-based phase-based method), and (c) the proposed method. The right panel shows a zoomed-in view of the red box in the left panel.
• Figure 5: Simultaneous recovery of multiple sound sources. (a) Experimental setup with two speakers playing tone of 100 Hz tone (left) and a 120 Hz tone (right). (b) ROI selection on an accumulated event image. Within the larger, manually selected region (green box), the final calculated ROI (left speaker: red box, right speaker: blue box) is automatically centered on the pixel with the highest event count. (c) The FFT of the signals recovered from each speaker's corresponding ROI. The resulting spectra clearly show distinct peaks at 100 Hz (red) and 120 Hz (blue), demonstrating the ability to individually record multiple sources from a single event stream.