EventCrab: Harnessing Frame and Point Synergy for Event-based Action Recognition and Beyond

TL;DR

This article presents a synergy-aware framework, i.e., EventCrab, that adeptly integrates the"lighter"frame-specific networks for dense event frames with the"heavier"point-specific networks for sparse event points, balancing accuracy and efficiency.

Abstract

Event-based Action Recognition (EAR) possesses the advantages of high-temporal resolution capturing and privacy preservation compared with traditional action recognition. Current leading EAR solutions typically follow two regimes: project unconstructed event streams into dense constructed event frames and adopt powerful frame-specific networks, or employ lightweight point-specific networks to handle sparse unconstructed event points directly. However, such two regimes are blind to a fundamental issue: failing to accommodate the unique dense temporal and sparse spatial properties of asynchronous event data. In this article, we present a synergy-aware framework, i.e., EventCrab, that adeptly integrates the "lighter" frame-specific networks for dense event frames with the "heavier" point-specific networks for sparse event points, balancing accuracy and efficiency. Furthermore, we establish a joint frame-text-point representation space to bridge distinct event frames and points. In specific, to better exploit the unique spatiotemporal relationships inherent in asynchronous event points, we devise two strategies for the "heavier" point-specific embedding: i) a Spiking-like Context Learner (SCL) that extracts contextualized event points from raw event streams. ii) an Event Point Encoder (EPE) that further explores event-point long spatiotemporal features in a Hilbert-scan way. Experiments on four datasets demonstrate the significant performance of our proposed EventCrab, particularly gaining improvements of 5.17% on SeAct and 7.01% on HARDVS.

Figures (7)

• Figure 2: Framework of the proposed EventCrab. For the event-point embedding, the Spiking-like Context Learner (SCL) and the Event-Point Encoder (EPE) are designed to extract contextual points and explore point features $\bm{f}_\mathrm{o}^\mathrm{e}$ with consideration of the spatiotemporal dependencies in asynchronous event points, respectively. It is guided by the point-prompt feature $\bm{f}_\mathrm{o}^\mathrm{t}$ from CLIP Text Encoder with the point-related prompt. Meanwhile, for the event-frame embedding, the event frames stacked from the event stream are fed to an Event-Frame Encoder (e.g., Transformer) to obtain the event-frame feature $\bm{f}_\mathrm{a}^\mathrm{e}$, which is similarly guided by the frame-prompt feature $\bm{f}_\mathrm{a}^\mathrm{t}$ from CLIP Text Encoder with the frame-related prompt.
• Figure 3: (left) Impact of different values of $\lambda$ balanced between event-frame and event-point on PAF dataset. (right) Impact of different numbers of Spiking Mamba blocks proposed in Event-Point Encoder on PAF dataset.
• Figure 4: Visualization of events before/after processed by SCL on the SeAct dataset.
• Figure 5: The t-SNE visualization of event features learned by different methods on the SeAct dataset. Ten action classes on the dataset are randomly selected. Best view in color.
• Figure 6: Visualization of the Top-3 predicted results on the SeAct dataset.