Segment Any Events with Language

TL;DR

SEAL tackles Open-Vocabulary Event Instance Segmentation by introducing Multimodal Hierarchical Semantic Guidance (MHSG) and a lightweight Multimodal Fusion Network built on a SAM-based event backbone. By leveraging hierarchical visual (semantic, instance, part) and textual priors within a unified framework, SEAL achieves open-vocabulary open-world event segmentation with high efficiency. The authors construct four OV-EIS benchmarks to evaluate label and semantic granularity, and show that SEAL outperforms baselines in both accuracy and inference speed, while maintaining a compact parameter footprint. The approach holds potential for robust, language-driven event understanding on edge devices, with planned extensions to spatiotemporal, prompt-free OV-EIS variants.

Abstract

Scene understanding with free-form language has been widely explored within diverse modalities such as images, point clouds, and LiDAR. However, related studies on event sensors are scarce or narrowly centered on semantic-level understanding. We introduce SEAL, the first Semantic-aware Segment Any Events framework that addresses Open-Vocabulary Event Instance Segmentation (OV-EIS). Given the visual prompt, our model presents a unified framework to support both event segmentation and open-vocabulary mask classification at multiple levels of granularity, including instance-level and part-level. To enable thorough evaluation on OV-EIS, we curate four benchmarks that cover label granularity from coarse to fine class configurations and semantic granularity from instance-level to part-level understanding. Extensive experiments show that our SEAL largely outperforms proposed baselines in terms of performance and inference speed with a parameter-efficient architecture. In the Appendix, we further present a simple variant of our SEAL achieving generic spatiotemporal OV-EIS that does not require any visual prompts from users in the inference. Check out our project page in https://0nandon.github.io/SEAL

Figures (14)

• Figure 1: a) Image-based models are vulnerable to severe image degradation, while event-based models remain robust by leveraging event inputs. b) Our SEAL effectively recognizes both part- and object-level instances on both noun- and sentence-level text queries. c) Our SEAL outperforms existing methods in performance and inference speed with paremeter-efficient architecture.
• Figure 2: Four Types of Baseline under Three Categories. The three categories (AR-CDG, Hybrid, AF-DA) are designed as a baseline based on the strategy of transferring image-domain knowledge to the event domain. Refer to Sec. \ref{['sec:preliminary']} and Supplementary material Sec. \ref{['sec:sup_baselines']} for further details.
• Figure 3: Overall framework of SEAL. The MHSG module (Red and Green) provides rich multimodal semantic guidance across multiple levels of granularity including part-level and instance-level. The multimodal fusion network of SEAL (Purple) enhances class prediction from a given binary mask by encoding rich semantic and spatial priors to produce a CLIP-aligned mask feature.
• Figure 4: Event feature space visualized by UMAP mcinnes2018umap. Event representation learned without SE and MFE exhibits dead masks (Red box) and semantic conflict (Purple box) with indistinct feature space. Our model addresses these issues by encoding spatial prior via SE and MFE to give a more discriminative feature space (Green box). Refer to Sec. \ref{['sec:abl']} for further details.
• Figure 5: (a) We first generate class-agnostic masks using SAM, and then assign semantic labels to each mask proposal either from ground-truth semantic maps or through human annotation. (b) Our four EIS benchmarks simulate label granularity from coarse to fine-grained class configurations and semantic granularity from instance-level to part-level understanding.