Exploring The Missing Semantics In Event Modality

TL;DR

Event cameras inherently lack static semantic information, making high-fidelity video reconstruction challenging. The authors propose Semantic-E2VID, a semantic-aware E2V framework that injects frame-level semantics into the event modality via Cross-Modal Feature Alignment (CFA) from the Segment Anything Model (SAM), a Semantic-aware Feature Fusion (SFF) block to integrate semantics into event features, and a semantic perceptual supervision to enforce semantically consistent reconstructions. The approach is complemented by temporal consistency and relational knowledge distillation losses, and is validated across six benchmarks with both referenced and non-referenced frames, achieving state-of-the-art results and sharper semantic structures. This work demonstrates the value of incorporating external semantic priors into event-based vision, enabling more accurate reconstructions and opening avenues for multi-task and real-world deployments.

Abstract

Event cameras offer distinct advantages such as low latency, high dynamic range, and efficient motion capture. However, event-to-video reconstruction (E2V), a fundamental event-based vision task, remains challenging, particularly for reconstructing and recovering semantic information. This is primarily due to the nature of the event camera, as it only captures intensity changes, ignoring static objects and backgrounds, resulting in a lack of semantic information in captured event modality. Further, semantic information plays a crucial role in video and frame reconstruction, yet is often overlooked by existing E2V approaches. To bridge this gap, we propose Semantic-E2VID, an E2V framework that explores the missing visual semantic knowledge in event modality and leverages it to enhance event-to-video reconstruction. Specifically, Semantic-E2VID introduces a cross-modal feature alignment (CFA) module to transfer the robust visual semantics from a frame-based vision foundation model, the Segment Anything Model (SAM), to the event encoder, while aligning the high-level features from distinct modalities. To better utilize the learned semantic feature, we further propose a semantic-aware feature fusion (SFF) block to integrate learned semantics in frame modality to form event representations with rich semantics that can be decoded by the event decoder. Further, to facilitate the reconstruction of semantic information, we propose a novel Semantic Perceptual E2V Supervision that helps the model to reconstruct semantic details by leveraging SAM-generated categorical labels. Extensive experiments demonstrate that Semantic-E2VID significantly enhances frame quality, outperforming state-of-the-art E2V methods across multiple benchmarks. The sample code is included in the supplementary material.

Figures (9)

• Figure 1: (a): A challenge of event-to-video reconstruction is: event camera only captures intensity changes, containing few semantic information, (b): video reconstruction task requires recovering the missing semantics. Thus, (c): using only events that contain few semantics as input leads to problems such as lacking semantic details. Our approach explores the missing semantics in event modality and utilizes them to produce more accurate results.
• Figure 2: Overview of our proposed framework. Given input events, we extract event features $\mathcal{F}_e$ using an encoder. To learn rich semantics from SAM without misleading the event encoder, the Cross-Modal Feature Alignment (CFA) module aligns $\mathcal{F}_e$ with semantics $\mathcal{F}_{sam}$, producing learned semantics $\mathcal{F}_{semantic}$. The Semantic-aware Feature Fusion (SFF) block further enhances $\mathcal{F}_e$ by leveraging $\mathcal{F}_{semantic}$, yielding event features with rich semantics $\mathcal{F}_e'$. A decoder reconstructs the final frame supervised by reconstruction and distillation losses.
• Figure 3: Illustration of our proposed Semantic Perceptual E2V Supervision. Instead of computing the perceptual loss uniformly over the entire image, we apply a binary categorical mask $\mathcal{M}$ derived from SAM’s segmentation outputs to the first-layer features of the perceptual network (AlexNet). This enhances supervision on semantically important regions (e.g., objects), while deeper layers retain full-frame guidance to preserve overall image quality.
• Figure 4: Qualitative comparisons on sequence datasets with referenced GT frames. Semantic-E2VID produces sharper textures, more accurate structural details, and fewer artifacts compared to other methods, particularly under challenging illumination and motion conditions. The integration of high-level semantic priors enables our approach to better reconstruct complex scenes and recover missing visual cues from sparse event data.
• Figure 5: Visual comparison for colorful frame reconstruction on the CED dataset. Our approach demonstrates better color fidelity and structural consistency, effectively recovering fine-grained details and natural color tones even under challenging conditions.