Decoupling Amplitude and Phase Attention in Frequency Domain for RGB-Event based Visual Object Tracking

TL;DR

This work tackles robust RGB–Event visual object tracking under challenging conditions by moving beyond conventional feature-level fusion to an early fusion strategy in the frequency domain. It introduces decoupled amplitude–phase attention to selectively inject high-frequency event information into RGB features and a motion-guided spatial sparsification module to focus on target-relevant regions, significantly reducing backbone computation. The approach, implemented with a Diff-FFT ViT and HiViT backbone, achieves strong results on FE108, FELT, and COESOT while maintaining real-time speed (27 FPS), demonstrating improved robustness to low-light and fast-motion scenarios. Overall, the method leverages event cameras’ high dynamic range and temporal resolution to deliver accurate and efficient RGB–Event tracking with practical impact for real-world surveillance and robotics.

Abstract

Existing RGB-Event visual object tracking approaches primarily rely on conventional feature-level fusion, failing to fully exploit the unique advantages of event cameras. In particular, the high dynamic range and motion-sensitive nature of event cameras are often overlooked, while low-information regions are processed uniformly, leading to unnecessary computational overhead for the backbone network. To address these issues, we propose a novel tracking framework that performs early fusion in the frequency domain, enabling effective aggregation of high-frequency information from the event modality. Specifically, RGB and event modalities are transformed from the spatial domain to the frequency domain via the Fast Fourier Transform, with their amplitude and phase components decoupled. High-frequency event information is selectively fused into RGB modality through amplitude and phase attention, enhancing feature representation while substantially reducing backbone computation. In addition, a motion-guided spatial sparsification module leverages the motion-sensitive nature of event cameras to capture the relationship between target motion cues and spatial probability distribution, filtering out low-information regions and enhancing target-relevant features. Finally, a sparse set of target-relevant features is fed into the backbone network for learning, and the tracking head predicts the final target position. Extensive experiments on three widely used RGB-Event tracking benchmark datasets, including FE108, FELT, and COESOT, demonstrate the high performance and efficiency of our method. The source code of this paper will be released on https://github.com/Event-AHU/OpenEvTracking

Figures (7)

• Figure 1: (a, b) Traditional Siamese and single-stream trackers need to process all visual tokens during the multimodal feature fusion and extraction stages, respectively, leading to high computational complexity. (c) Our framework proposes decoupled amplitude and phase attention to halve the token count in early fusion, and uses motion-guided spatial sparsification to focus on target-relevant tokens, substantially reducing backbone computation.
• Figure 2: An overview of our proposed Amplitude–Phase attention and Motion-guided sparsification framework for efficient RGB-Event tracking, called APMTrack. Specifically, RGB and event inputs are first decoupled into amplitude and phase in the frequency domain, allowing high-frequency event information to enhance RGB modality via amplitude and phase attention. The event encoder extracts motion cues, which are refined by the FFT-based differential ViT, and subsequently guide a spatial sparsification module for adaptive Top-$K$ token selection. The selected search tokens, combined with template features, are processed by the backbone, and the tracking head predicts the final target location.
• Figure 3: Tracking results (SR) under each challenging factor.
• Figure 4: Visualization results of our early fusion and motion-guided spatial sparsification strategy.
• Figure 5: Visualization of the attention activation maps generated by our method.