Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognition

Zongyou Yu, Qiang Qu, Xiaoming Chen, Chen Wang

TL;DR

This work tackles pure zero-shot event-based recognition, addressing the gap where CLIP-based approaches require training and struggle with event-domain gaps. It proposes converting raw event streams into frames (event frames or reconstructed frames) and using large language models with carefully engineered prompts to perform recognition without fine-tuning. Key findings show GPT-4o yields the strongest performance across three datasets, often outperforming state-of-the-art zero-shot methods by large margins, with input representation and prompt design modulating gains; reconstructed frames help certain datasets while others favor direct event frames due to resolution and noise considerations. The results indicate a promising new direction for cross-modal event understanding and provide benchmarks for future work.

Abstract

Recent advancements in event-based zero-shot object recognition have demonstrated promising results. However, these methods heavily depend on extensive training and are inherently constrained by the characteristics of CLIP. To the best of our knowledge, this research is the first study to explore the understanding capabilities of large language models (LLMs) for event-based visual content. We demonstrate that LLMs can achieve event-based object recognition without additional training or fine-tuning in conjunction with CLIP, effectively enabling pure zero-shot event-based recognition. Particularly, we evaluate the ability of GPT-4o / 4turbo and two other open-source LLMs to directly recognize event-based visual content. Extensive experiments are conducted across three benchmark datasets, systematically assessing the recognition accuracy of these models. The results show that LLMs, especially when enhanced with well-designed prompts, significantly improve event-based zero-shot recognition performance. Notably, GPT-4o outperforms the compared models and exceeds the recognition accuracy of state-of-the-art event-based zero-shot methods on N-ImageNet by five orders of magnitude. The implementation of this paper is available at \url{https://github.com/ChrisYu-Zz/Pure-event-based-recognition-based-LLM}.

Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognition

TL;DR

This work tackles pure zero-shot event-based recognition, addressing the gap where CLIP-based approaches require training and struggle with event-domain gaps. It proposes converting raw event streams into frames (event frames or reconstructed frames) and using large language models with carefully engineered prompts to perform recognition without fine-tuning. Key findings show GPT-4o yields the strongest performance across three datasets, often outperforming state-of-the-art zero-shot methods by large margins, with input representation and prompt design modulating gains; reconstructed frames help certain datasets while others favor direct event frames due to resolution and noise considerations. The results indicate a promising new direction for cross-modal event understanding and provide benchmarks for future work.

Abstract

Recent advancements in event-based zero-shot object recognition have demonstrated promising results. However, these methods heavily depend on extensive training and are inherently constrained by the characteristics of CLIP. To the best of our knowledge, this research is the first study to explore the understanding capabilities of large language models (LLMs) for event-based visual content. We demonstrate that LLMs can achieve event-based object recognition without additional training or fine-tuning in conjunction with CLIP, effectively enabling pure zero-shot event-based recognition. Particularly, we evaluate the ability of GPT-4o / 4turbo and two other open-source LLMs to directly recognize event-based visual content. Extensive experiments are conducted across three benchmark datasets, systematically assessing the recognition accuracy of these models. The results show that LLMs, especially when enhanced with well-designed prompts, significantly improve event-based zero-shot recognition performance. Notably, GPT-4o outperforms the compared models and exceeds the recognition accuracy of state-of-the-art event-based zero-shot methods on N-ImageNet by five orders of magnitude. The implementation of this paper is available at \url{https://github.com/ChrisYu-Zz/Pure-event-based-recognition-based-LLM}.
Paper Structure (7 sections, 3 figures, 1 table)

This paper contains 7 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets and Task Setups
  4. Task Setups
  5. Experiments
  6. Results
  7. Conclusion

Figures (3)

  • Figure 1: Comparison of methods for event-based object recognition. Unlike existing zero-shot approaches, our pure zero-shot recognition using LLMs eliminates the need for additional training, such as event encoder training or CLIP fine-tuning, while addressing the inherent limitations of CLIP.
  • Figure 2: Workflow for event-based visual content recognition using LLMs. The process starts by converting raw event stream into event frames, or alternatively, into reconstructed frames using b7 and b8. These converted frames, along with a prompt, are then input into the LLM for recognition. Once the model returns its results, post-processing is performed to obtain the final outcome.
  • Figure 3: Comparison of LLMs' understanding of event-based visual content and traditional visual content. A contrast between the event-based visual content recognition abilities of the evaluated LLMs and the traditional visual content perception abilities of the models assessed in b9.