An Efficient and Effective Encoder Model for Vision and Language Tasks in the Remote Sensing Domain

TL;DR

The paper introduces GeoMELT, a compact 271M-parameter multimodal encoder for remote sensing that jointly handles text generation and cross-modal retrieval. Built on a BEiT-3 foundation with Multiway Transformer blocks, it supports vision, language, and joint representations within a single architecture, enabling it to function as a vision encoder, a dual encoder for retrieval, or a cross-encoder for generation. It achieves competitive or state-of-the-art results across captioning, VQA, grounding, retrieval, and zero-shot classification on RS benchmarks while requiring far less compute than large LVLMs, aided by a two-stage training regimen and dynamic loss balancing. The work highlights practical efficiency gains for edge deployment and broad accessibility to RS-VL capabilities, with future directions including multisensor fusion, composed retrieval, and integration with LVLMs for retrieval-augmented generation and reasoning.

Abstract

The remote sensing community has recently seen the emergence of methods based on Large Vision and Language Models (LVLMs) that can address multiple tasks at the intersection of computer vision and natural language processing. To fully exploit the potential of such models, a significant focus has been given to the collection of large amounts of training data that cover multiple remote sensing-specific tasks, such as image captioning or visual question answering. However, the cost of using and training LVLMs is high, due to the large number of parameters. While multiple parameter-efficient adaptation techniques have been explored, the computational costs of training and inference with these models can remain prohibitive for most institutions. In this work, we explore the use of encoder-only architectures and propose a model that can effectively address multi-task learning while remaining compact in terms of the number of parameters. In particular, our model tackles combinations of tasks that are not typically explored in a unified model: the generation of text from remote sensing images and cross-modal retrieval. The results of our GeoMELT model - named from Multi-task Efficient Learning Transformer - in established benchmarks confirm the efficacy and efficiency of the proposed approach.

Figures (5)

• Figure 1: GeoMELT is a multi-task vision and language model for remote sensing. Across image captioning and cross-modal retrieval evaluatuions, each aggregated over multiple datasets, GeoMELT consistently achieves top overall results while staying efficient regarding the number of parameters.
• Figure 2: Overview of the GeoMELT model based on a multimodal encoder architecture with a total of 271M parameters. The earlier layers contain a shared Multi-Head Self-Attention (MHSA) layer and Feed-Forward Networks (FFN) that are specific to each modality. The top layers contain a shared FFN. Masked language modeling is used for text generation training. Importantly, text generation with GeoMELT can go beyond image captioning and include visual grounding and visual question answering tasks. For retrieval tasks, the model can be used as a dual encoder and trained with contrastive learning.
• Figure 3: Examples of image–text retrieval. In some cases, GeoMELT produces results that are more semantically aligned than the ground truth. In the first row, the ground-truth image does not contain the presence of multiple airplanes, which GeoMELT correctly identifies. In the last row, GeoMELT obtains a more detailed and contextually rich description of the image.
• Figure 4: Examples of visual grounding outputs given textual queries. GeoMELT can localize the objects of interest and successfully differentiate them based on localization.
• Figure 5: Examples of GeoMELT outputs across different tasks, including captioning, long captioning, visual question answering, and visual grounding. GeoMELT produces concise descriptions, generates detailed long captions, accurately answers questions from RSVQA datasets, and successfully identifies queried objects.