EvMic: Event-based Non-contact sound recovery from effective spatial-temporal modeling

TL;DR

This work addresses non-contact sound recovery by leveraging the high temporal resolution and sparsity of event cameras, augmented with a laser-matrix to amplify surface gradients. It introduces EvMic, a Blender-based synthetic dataset, and a three-part network that combines sparse convolution, a spatial aggregation block, and a Mamba-based temporal model to recover audio from visual vibrations. Experimental results on synthetic and real-world data show improved SNR and intelligibility over baselines, demonstrating the feasibility of high-frequency sound recovery with event-based vision. The approach offers a scalable framework for non-contact audio reconstruction with potential applications in surveillance and material-property analysis, while acknowledging sim-to-real gaps and lighting-gradient dependencies as future challenges.

Abstract

When sound waves hit an object, they induce vibrations that produce high-frequency and subtle visual changes, which can be used for recovering the sound. Early studies always encounter trade-offs related to sampling rate, bandwidth, field of view, and the simplicity of the optical path. Recent advances in event camera hardware show good potential for its application in visual sound recovery, because of its superior ability in capturing high-frequency signals. However, existing event-based vibration recovery methods are still sub-optimal for sound recovery. In this work, we propose a novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal information from the event stream. We first generate a large training set using a novel simulation pipeline. Then we designed a network that leverages the sparsity of events to capture spatial information and uses Mamba to model long-term temporal information. Lastly, we train a spatial aggregation block to aggregate information from different locations to further improve signal quality. To capture event signals caused by sound waves, we also designed an imaging system using a laser matrix to enhance the gradient and collected multiple data sequences for testing. Experimental results on synthetic and real-world data demonstrate the effectiveness of our method.

Figures (11)

• Figure 1: Illustration of our event-based non-contact sound recovery. We try to recover sound from the visual vibration of the object caused by the sound wave. Compared with the traditional high-speed camera solution (top), we proposed to use an event camera to capture a temporally dense signal (bottom). We first utilize a laser matrix (left) to amplify the gradient and an event camera to capture the vibrations. Then, our learning-based approach to spatial-temporal modeling enables us to recover better signals.
• Figure 2: (a) Our data simulation starts with controlling the objects' vibration. We utilize audio to manipulate the coordinates of objects resulting in their vibrations across random directions. Then we use an event simulator to generate the corresponding events. The generated events are used for training. (b) The synthetic vibrating speckles are used for fine-tuning and testing.
• Figure 3: (a) Overview of our proposed network architecture. The event stream is processed into voxel grids, from which patches centered around the speckles are selected. First, the patches are input into a sparse convolution-based lightweight backbone to extract visual features. Next, a spatial attention block aggregates the information in the different patches. Finally, Mamba is employed to model long-term temporal information and reconstruct the audio that caused the object’s vibration. (b) and (c) illustrate the detailed structure of SAB and SSM. (c) At time t $g_t$ is the input feature, $o_t$ is the output and $h_t$ denotes the hidden state. A, B, and C are the gating weights optimized by Mamba. $\Delta$ is used to discretize the continuous parameters $A$ and $B$.
• Figure 4: Qualitative comparison results on the real-world data of a chipbag. Audio is provided in the supplementary.
• Figure 5: Qualitative comparison results on the real-world data of a speaker. Audio is provided in the supplementary.