Leveraging Lightweight Entity Extraction for Scalable Event-Based Image Retrieval

TL;DR

This work tackles scalable image retrieval from long, entity-rich natural language queries by introducing a lightweight two-stage pipeline that first filters candidates via event-guided entity extraction and BM25, then re ranks with dual BEiT-3 multimodal models. By combining symbolic entity filtering with long-form vision-language representations and Reciprocal Rank Fusion, the approach achieves strong OpenEvents v1 performance (mAP $0.559$ on the public split, significantly higher than the baseline $0.323$). The methodology includes entity-aware indexing in Elasticsearch, offline BEiT-3 embeddings stored in a vector database, and a two-branch reranking strategy that balances literal event cues and deeper semantic alignment. The results demonstrate practical, scalable retrieval for real-world, event-driven image search, with future directions including more advanced event extraction and retrieval-augmented techniques to further boost accuracy and generalization.

Abstract

Retrieving images from natural language descriptions is a core task at the intersection of computer vision and natural language processing, with wide-ranging applications in search engines, media archiving, and digital content management. However, real-world image-text retrieval remains challenging due to vague or context-dependent queries, linguistic variability, and the need for scalable solutions. In this work, we propose a lightweight two-stage retrieval pipeline that leverages event-centric entity extraction to incorporate temporal and contextual signals from real-world captions. The first stage performs efficient candidate filtering using BM25 based on salient entities, while the second stage applies BEiT-3 models to capture deep multimodal semantics and rerank the results. Evaluated on the OpenEvents v1 benchmark, our method achieves a mean average precision of 0.559, substantially outperforming prior baselines. These results highlight the effectiveness of combining event-guided filtering with long-text vision-language modeling for accurate and efficient retrieval in complex, real-world scenarios. Our code is available at https://github.com/PhamPhuHoa-23/Event-Based-Image-Retrieval