SpikeMM: Flexi-Magnification of High-Speed Micro-Motions
Baoyue Zhang, Yajing Zheng, Shiyan Chen, Jiyuan Zhang, Kang Chen, Zhaofei Yu, Tiejun Huang
TL;DR
SpikeMM addresses the challenge of magnifying high-speed micro-motions by leveraging spike cameras, which capture asynchronous spike streams at high temporal rates, to overcome motion blur and sampling limitations of conventional cameras. It introduces a self-supervised framework combining multi-level information extraction (two window lengths with blind-spot networks), spatial upsampling via implicit neural representations, and integration with learning-based motion magnification algorithms. Key contributions include the first spike-based self-supervised motion magnification method, a multi-level spike representation with BSN-based noise handling, and an INR-based super-resolution branch that enables arbitrary-scale magnification, validated on a spike-camera dataset with four real-world scenes. The approach offers practical impact for high-speed fault detection, fluid mechanics analysis, medical dynamics monitoring, and security surveillance where traditional imaging falls short.
Abstract
The amplification of high-speed micro-motions holds significant promise, with applications spanning fault detection in fast-paced industrial environments to refining precision in medical procedures. However, conventional motion magnification algorithms often encounter challenges in high-speed scenarios due to low sampling rates or motion blur. In recent years, spike cameras have emerged as a superior alternative for visual tasks in such environments, owing to their unique capability to capture temporal and spatial frequency domains with exceptional fidelity. Unlike conventional cameras, which operate at fixed, low frequencies, spike cameras emulate the functionality of the retina, asynchronously capturing photon changes at each pixel position using spike streams. This innovative approach comprehensively records temporal and spatial visual information, rendering it particularly suitable for magnifying high-speed micro-motions.This paper introduces SpikeMM, a pioneering spike-based algorithm tailored specifically for high-speed motion magnification. SpikeMM integrates multi-level information extraction, spatial upsampling, and motion magnification modules, offering a self-supervised approach adaptable to a wide range of scenarios. Notably, SpikeMM facilitates seamless integration with high-performance super-resolution and motion magnification algorithms. We substantiate the efficacy of SpikeMM through rigorous validation using scenes captured by spike cameras, showcasing its capacity to magnify motions in real-world high-frequency settings.