Cross-Modal Binary Attention: An Energy-Efficient Fusion Framework for Audio-Visual Learning

TL;DR

The paper tackles the challenge of energy efficient, scalable cross modal fusion for audio visual data by introducing CMQKA, a cross modal Query-Key attention mechanism that achieves linear $O(N)$ complexity using binary spike operations. Built on CMQKA, the SNNergy framework implements a three stage hierarchical spiking transformer with a SPDS downsampling module, enabling multi scale cross modal interaction while preserving modality specific discriminative power through residual fusion. Empirical results on AVE, CREMA-D, and UrbanSound8K-AV demonstrate state of the art performance among SNN baselines and substantial efficiency benefits, validating linear complexity cross modal attention for hierarchical multimodal learning. The work highlights practical potential for real time edge deployment on neuromorphic hardware, offering a path toward energy efficient, high accuracy audio visual intelligence in resource constrained environments.

Abstract

Effective multimodal fusion requires mechanisms that can capture complex cross-modal dependencies while remaining computationally scalable for real-world deployment. Existing audio-visual fusion approaches face a fundamental trade-off: attention-based methods effectively model cross-modal relationships but incur quadratic computational complexity that prevents hierarchical, multi-scale architectures, while efficient fusion strategies rely on simplistic concatenation that fails to extract complementary cross-modal information. We introduce CMQKA, a novel cross-modal fusion mechanism that achieves linear O(N) complexity through efficient binary operations, enabling scalable hierarchical fusion previously infeasible with conventional attention. CMQKA employs bidirectional cross-modal Query-Key attention to extract complementary spatiotemporal features and uses learnable residual fusion to preserve modality-specific characteristics while enriching representations with cross-modal information. Building upon CMQKA, we present SNNergy, an energy-efficient multimodal fusion framework with a hierarchical architecture that processes inputs through progressively decreasing spatial resolutions and increasing semantic abstraction. This multi-scale fusion capability allows the framework to capture both local patterns and global context across modalities. Implemented with event-driven binary spike operations, SNNergy achieves remarkable energy efficiency while maintaining fusion effectiveness and establishing new state-of-the-art results on challenging audio-visual benchmarks, including CREMA-D, AVE, and UrbanSound8K-AV, significantly outperforming existing multimodal fusion baselines. Our framework advances multimodal fusion by introducing a scalable fusion mechanism that enables hierarchical cross-modal integration with practical energy efficiency for real-world audio-visual intelligence systems.

Figures (7)

• Figure 1: Illustration of the Complementary Spatial Attention pathway within the CMQKA module: Video spikes $\mathbf{V}' \in \{0,1\}^{T \times B \times C \times N}$ and audio spikes $\mathbf{A}' \in \{0,1\}^{T \times B \times C \times N}$ are transformed through learnable projections to generate Query $\mathbf{Q}_v$ and Key $\mathbf{K}_a$, both in $\{0,1\}^{T \times B \times C \times N}$. For multi-head attention, Q is reshaped to $\{0,1\}^{T \times B \times h \times C_h \times N}$ where $h$ is the number of heads and $C_h = C/h$ is the per-head channel dimension. The Query is summed over the per-head channel dimension $C_h$ and passed through a spiking neuron (SN) to produce a binary spatial attention tensor $\mathbf{A}^{(s)} \in \{0,1\}^{T \times B \times h \times 1 \times N}$. This attention tensor selectively masks the Key matrix through element-wise multiplication ($\mathbf{S}^{v \leftarrow a} = \mathbf{A}_t^{(s)} \odot \mathbf{K}_a$) to extract complementary spatial features from the audio modality that are relevant to the video modality. The mechanism maintains $O(N)$ linear complexity through binary spike operations, avoiding the quadratic cost of traditional self-attention.
• Figure 2: Illustration of the Complementary Temporal Attention pathway within the CMQKA module: The spatially-flattened inputs $\mathbf{V}', \mathbf{A}' \in \{0,1\}^{T \times C \times N}$ are first permuted to $N \times C \times T$ to operate along the temporal dimension. The permuted inputs are transformed through learnable projections to generate Query $\mathbf{Q}_v$ from video and Key $\mathbf{K}_a$ from audio, both in $\{0,1\}^{N \times C \times T}$ and then permuted back to $T \times C \times N$. The Query matrix is summed over channels and passed through a spiking neuron to produce a binary temporal attention vector $\mathbf{A}^{(t)} \in \{0,1\}^{T \times 1 \times N}$. This attention vector selectively masks the Key matrix through element-wise multiplication ($\mathbf{T}^{v \leftarrow a} = \mathbf{A}_t^{(t)} \odot \mathbf{K}_a^{(t)}$) to extract complementary temporal features $\mathbf{T}^{v \leftarrow a}$ from the audio modality that capture temporal patterns relevant to the video modality. The mechanism maintains $O(N)$ linear complexity, enabling efficient temporal cross-modal feature extraction.
• Figure 3: Fusion and residual integration within the CMQKA module: The complementary spatial features $\mathbf{S}^{v \leftarrow a}$ and temporal features $\mathbf{T}^{v \leftarrow a}$ are pooled along their respective dimensions to produce reduced representations $\mathbf{S}_{\text{red}}^{v \leftarrow a}$ and $\mathbf{T}_{\text{red}}^{v \leftarrow a}$. Element-wise multiplication combines these reduced features into joint spatiotemporal complementary features $\mathbf{H}^{v \leftarrow a}$. A learnable residual connection integrates these cross-modal features into the original video representation $\mathbf{V}$, yielding enriched output features $\tilde{\mathbf{V}}$. A spiking MLP with residual connection further refines the fused features to produce the final output $\mathbf{V}_{\text{out}}$. This fusion strategy preserves modality-specific characteristics while enhancing representations with cross-modal complementary information.
• Figure 4: FLOPs comparison across stages 1 and 2 on both CREMA-D and AVE datasets, demonstrating the computational efficiency of CMQKA's linear-complexity fusion mechanism. The gap between quadratic and linear attention grows with spatial resolution, validating the scalability benefits of our approach across diverse audio-visual benchmarks.
• Figure 5: Runtime comparison across stages 1 and 2 on both CREMA-D and AVE datasets, showing the inference speed advantage of linear-complexity CMQKA. Faster execution enables real-time multimodal processing for latency-sensitive applications.