EventGPT: Event Stream Understanding with Multimodal Large Language Models

TL;DR

EventGPT addresses event stream understanding by bridging neuromorphic camera data with language models. It introduces a modular architecture (event encoder, spatio-temporal aggregator, linear projector, event-language adapter, and a large language model) trained via a three-stage paradigm to progressively align sparse event signals with language. Two large datasets, N-ImageNet-Chat (>1e6 synthetic samples) and Event-Chat (≈59k instruction entries), support cross-modal learning and comprehensive evaluation. Experiments show EventGPT surpasses prior MLLMs on generation quality, descriptive accuracy, reasoning, and downstream open-set tasks, demonstrating robust performance in low-light and high-speed conditions.

Abstract

Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, to the best of our knowledge, marking a pioneering attempt to integrate large language models (LLMs) with event stream comprehension. To mitigate the huge domain gaps, we develop a three-stage optimization paradigm to gradually equip a pre-trained LLM with the capability of understanding event-based scenes. Our EventGPT comprises an event encoder, followed by a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, RGB image-text pairs generated by GPT are leveraged to warm up the linear projector, referring to LLaVA, as the gap between natural image and language modalities is relatively smaller. Secondly, we construct a synthetic yet large dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, Event-Chat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability.

Figures (9)

• Figure 1: Our EventGPT is the first multimodal large language model tailored for event stream understanding, including scene summarization, reasoning, and question answering, with a great potential for downstream tasks.
• Figure 2: Overview of our framework. The event encoder transforms raw event tensors into high-dimensional features, which are then aggregated by a spatio-temporal module. These representations are projected and aligned with the large language model, enabling nuanced understanding of event streams and supporting various downstream applications.
• Figure 3: Data distribution across three progressive training stages in our EventGPT containing image-language alignment, event-language alignment, and instruction tuning.
• Figure 4: Instruction examples from the Event-Chat dataset for multimodal task prompts and responses in image captioning, visual reasoning, and visual question answering.
• Figure 5: Our three-stage pipeline consists of (1) training a linear projector to achieve visual-language alignment, (2) incorporating a spatio-temporal aggregator and an event-language adapter to enable event-language alignment, and (3) performing full fine-tuning of all modules to facilitate cohesive multimodal learning.