NewsRECON: News article REtrieval for image CONtextualization

TL;DR

NewsRECON introduces a RIS-agnostic framework that retrieves news articles to infer the date and location of an image by leveraging a large article corpus. The method combines a bi-encoder for initial retrieval with two specialized cross-encoders for location and event re-ranking, followed by clustering and an event-focused reranking, producing ranked article sets used as evidence for downstream date-location prediction. Empirical results on the TARA and 5Pils-OOC benchmarks show NewsRECON outperforms prior retrieval-based baselines and, when combined with an external MLLM, achieves new state-of-the-art performance in the absence of RIS evidence, while generalizing to out-of-distribution data. The work demonstrates the practical value of large-scale news corpora for image contextualization, offering a scalable tool for journalists and fact-checkers to verify multimodal evidence and debunk misinformation, with open-source code and a detailed analysis of ablations and corpus effects.

Abstract

Identifying when and where a news image was taken is crucial for journalists and forensic experts to produce credible stories and debunk misinformation. While many existing methods rely on reverse image search (RIS) engines, these tools often fail to return results, thereby limiting their practical applicability. In this work, we address the challenging scenario where RIS evidence is unavailable. We introduce NewsRECON, a method that links images to relevant news articles to infer their date and location from article metadata. NewsRECON leverages a corpus of over 90,000 articles and integrates: (1) a bi-encoder for retrieving event-relevant articles; (2) two cross-encoders for reranking articles by location and event consistency. Experiments on the TARA and 5Pils-OOC show that NewsRECON outperforms prior work and can be combined with a multimodal large language model to achieve new SOTA results in the absence of RIS evidence. We make our code available.

Figures (11)

• Figure 1: NewsRECON retrieves articles that are location or event-relevant, given an image as query.
• Figure 2: Overview of NewsRECON. Parts a to c illustrate the three training stages with images from TARA fu-etal-2022-theres. Part d shows the retrieval pipeline at inference time.
• Figure 3: News articles corpus creation process.
• Figure 4: Results on 5Pils-OOC (%). "All images" refers to the entire dataset (N=624), while "Images w/o RIS evidence" corresponds to the subset where RIS engines do not return any webpages as evidence (N=172).
• Figure 5: Predictions of different models for four instances of the TARA test set.