Time-step Mixup for Efficient Spiking Knowledge Transfer from Appearance to Event Domain

TL;DR

This work tackles the challenge of transferring semantic knowledge from RGB appearance data to the event domain for spiking neural networks (SNNs) by introducing Time-step Mixup Knowledge Transfer (TMKT). TMKT leverages a two-stream, time-step mixing design to create a common representation space and employs domain-alignment (CKA-based), Modality-Aware Guidance (MAG), and Mixup Ratio Perception (MRP) losses to smooth cross-modal learning. Experiments on N-Caltech101, CEP-DVS, and N-Omniglot demonstrate state-of-the-art or strong competitive performance, with ablations showing substantial gains from time-step mixing and the auxiliary losses. The approach yields better cross-modal interpretability and convergence, highlighting the practical potential of TMKT for energy-efficient, cross-modal SNN vision tasks.

Abstract

The integration of event cameras and spiking neural networks holds great promise for energy-efficient visual processing. However, the limited availability of event data and the sparse nature of DVS outputs pose challenges for effective training. Although some prior work has attempted to transfer semantic knowledge from RGB datasets to DVS, they often overlook the significant distribution gap between the two modalities. In this paper, we propose Time-step Mixup knowledge transfer (TMKT), a novel fine-grained mixing strategy that exploits the asynchronous nature of SNNs by interpolating RGB and DVS inputs at various time-steps. To enable label mixing in cross-modal scenarios, we further introduce modality-aware auxiliary learning objectives. These objectives support the time-step mixup process and enhance the model's ability to discriminate effectively across different modalities. Our approach enables smoother knowledge transfer, alleviates modality shift during training, and achieves superior performance in spiking image classification tasks. Extensive experiments demonstrate the effectiveness of our method across multiple datasets. The code will be released after the double-blind review process.

Figures (4)

• Figure 1: Different strategies for leveraging appearance data to assist spiking neural networks in learning the event domain. (a) Finetune training suffers from domain mismatch issues. (b) Domain alignment helps mitigate this issue to some extent. (c) Our Time-step Mixup offers a smoother learning paradigm, alleviating the convergence difficulties that arise during domain transition.
• Figure 2: The overview of our proposed Time-step Mixup Knowledge Transfer (TMKT) framework. TMKT employs a Time-step Mixup (TSM) strategy and introduces two auxiliary labels: a modality-aware guidance label and a mixup ratio label to enhance the supervision of temporal knowledge transfer. Both the event stream and the Time-step Mixup stream are fed into the network simultaneously, sharing all weights except for the final layer. Membrane potentials from the penultimate layer are used for domain alignment.
• Figure 3: Class Activation Mapping of Caltech101 and N-Caltech101. For each class, the top row shows static images, and the bottom row presents event data integrated into frames. Within each class, from left to right are: original input, baseline result, and the result of our method.
• Figure 4: Visualization of the loss landscapes for our method and the baseline on CEP-DVS dataset.