Multilingual Training-Free Remote Sensing Image Captioning

TL;DR

The paper presents a training-free, multilingual approach to remote sensing image captioning that leverages retrieval-augmented prompting. By encoding images with a domain-adapted SigLIP2, retrieving related captions and few-shot examples, and re-ranking via personalized PageRank, the method builds rich prompts for multilingual LLMs or VLMs to generate captions directly in target languages. Experiments across four datasets and nine languages demonstrate competitive results with supervised English models, with PageRank re-ranking yielding significant gains and direct multilingual generation outperforming translation-based pipelines. The work highlights the critical roles of domain-specific retrieval quality and coherent prompt construction in cross-lingual multimodal captioning for Earth observation data.

Abstract

Remote sensing image captioning has advanced rapidly through encoder--decoder models, although the reliance on large annotated datasets and the focus on English restricts global applicability. To address these limitations, we propose the first training-free multilingual approach, based on retrieval-augmented prompting. For a given aerial image, we employ a domain-adapted SigLIP2 encoder to retrieve related captions and few-shot examples from a datastore, which are then provided to a language model. We explore two variants: an image-blind setup, where a multilingual Large Language Model (LLM) generates the caption from textual prompts alone, and an image-aware setup, where a Vision--Language Model (VLM) jointly processes the prompt and the input image. To improve the coherence of the retrieved content, we introduce a graph-based re-ranking strategy using PageRank on a graph of images and captions. Experiments on four benchmark datasets across ten languages demonstrate that our approach is competitive with fully supervised English-only systems and generalizes to other languages. Results also highlight the importance of re-ranking with PageRank, yielding up to 35% improvements in performance metrics. Additionally, it was observed that while VLMs tend to generate visually grounded but lexically diverse captions, LLMs can achieve stronger BLEU and CIDEr scores. Lastly, directly generating captions in the target language consistently outperforms other translation-based strategies. Overall, our work delivers one of the first systematic evaluations of multilingual, training-free captioning for remote sensing imagery, advancing toward more inclusive and scalable multimodal Earth observation systems.

Figures (10)

• Figure 1: A general illustration for the proposed image captioning approach. For a given input image, the system first retrieves related captions, plus the most similar images and, for each of the images, also the corresponding most similar captions. The retrieved information is then assembled into a prompt, which can either be provided alone to a multilingual LLM (solid line) or, together with the input image, to a multilingual VLM (dashed line). In both cases, the model generates a caption in the target language. The prompt illustrated in the figure is a simplified example.
• Figure 2: An illustration for how the PageRank algorithm influences the final composition of the prompt. On the left, the weighting of the elements is based solely on SigLIP similarities (i.e., the original ranking). On the right, after PageRank propagation, the elements are re-ranked into a different distribution of node weights. For simplicity, this representation considers fewer elements and final retrieved captions compared to the experiments reported in the paper.
• Figure 3: Prompt for image captioning from the retrieved results. The green text between brackets corresponds to the changes made when using the image-aware strategy, in order to take into account the input image.
• Figure 4: Examples of generated captions from our models, for the English language and using the input image, the retrieved captions for the input image, and the few-shot examples, with re-ranking through personalized PageRank.
• Figure 5: Example for the positive and negative effects of using VLMs. On the first case, the input image helps the model clarifying that there are "three" airplanes on the photo, being rewarded by the references. On the other hand, on the second case, the VLM describes in too much detail the image, becoming less aligned with references and having lower scores.