Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge-aware Visual Question Generation for Remote Sensing Images

Siran Li, Li Mi, Javiera Castillo-Navarro, Devis Tuia

TL;DR

This work tackles knowledge-enriched visual question generation for remote sensing images by introducing KRSVQG, a BLIP-based vision-language model that ingests an image $I$ and an external knowledge sentence $S$ to produce a grounded, informative question $Q_{hat}$ via a captioning intermediary. The model comprises four modules (image encoder, caption decoder, text encoder, question decoder) and a three-stage training strategy, leveraging a knowledge triplet $T$ from ConceptNet to ground questions in both image content and domain knowledge. Two new datasets, NWPU-300 and TextRS-300, are constructed by pairing RS images with knowledge sentences and questions, enabling evaluation with standard VQG metrics; results show substantial improvements over strong baselines on BLEU, METEOR, ROUGE-L, and CIDEr. Overall, the approach advances knowledge-aware VQG for remote sensing, enabling more diverse and context-rich questions that can bolster VQA and visual dialogue systems in Earth observation contexts.

Abstract

With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing image retrieval. However, automatically generated image-based questions tend to be simplistic and template-based, which hinders the real deployment of question answering or visual dialogue systems. To enrich and diversify the questions, we propose a knowledge-aware remote sensing visual question generation model, KRSVQG, that incorporates external knowledge related to the image content to improve the quality and contextual understanding of the generated questions. The model takes an image and a related knowledge triplet from external knowledge sources as inputs and leverages image captioning as an intermediary representation to enhance the image grounding of the generated questions. To assess the performance of KRSVQG, we utilized two datasets that we manually annotated: NWPU-300 and TextRS-300. Results on these two datasets demonstrate that KRSVQG outperforms existing methods and leads to knowledge-enriched questions, grounded in both image and domain knowledge.

Knowledge-aware Visual Question Generation for Remote Sensing Images

TL;DR

This work tackles knowledge-enriched visual question generation for remote sensing images by introducing KRSVQG, a BLIP-based vision-language model that ingests an image and an external knowledge sentence to produce a grounded, informative question via a captioning intermediary. The model comprises four modules (image encoder, caption decoder, text encoder, question decoder) and a three-stage training strategy, leveraging a knowledge triplet from ConceptNet to ground questions in both image content and domain knowledge. Two new datasets, NWPU-300 and TextRS-300, are constructed by pairing RS images with knowledge sentences and questions, enabling evaluation with standard VQG metrics; results show substantial improvements over strong baselines on BLEU, METEOR, ROUGE-L, and CIDEr. Overall, the approach advances knowledge-aware VQG for remote sensing, enabling more diverse and context-rich questions that can bolster VQA and visual dialogue systems in Earth observation contexts.

Abstract

With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing image retrieval. However, automatically generated image-based questions tend to be simplistic and template-based, which hinders the real deployment of question answering or visual dialogue systems. To enrich and diversify the questions, we propose a knowledge-aware remote sensing visual question generation model, KRSVQG, that incorporates external knowledge related to the image content to improve the quality and contextual understanding of the generated questions. The model takes an image and a related knowledge triplet from external knowledge sources as inputs and leverages image captioning as an intermediary representation to enhance the image grounding of the generated questions. To assess the performance of KRSVQG, we utilized two datasets that we manually annotated: NWPU-300 and TextRS-300. Results on these two datasets demonstrate that KRSVQG outperforms existing methods and leads to knowledge-enriched questions, grounded in both image and domain knowledge.
Paper Structure (10 sections, 2 equations, 4 figures, 1 table)

This paper contains 10 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model Architecture
  4. Loss Functions
  5. Datasets
  6. Experimental Setup
  7. Baseline Methods and Evaluation Metrics
  8. Implementation Details
  9. Results
  10. Conclusion

Figures (4)

  • Figure 1: An example of image-based questions and a knowledge-aware question. In the questions, text from the image description is highlighted in green and text from external knowledge in orange.
  • Figure 2: The pipeline of our KRSVQG model comprises four components: an image encoder, a caption decoder, a text encoder, and a question decoder. With an image ($I$) and a knowledge sentence ($S$) as inputs, the model generates a knowledge-aware question ($\widehat{Q}$) based on the caption ($\widehat{C}$) and the knowledge sentence ($S$). $\oplus$ means addition and normalization.
  • Figure 3: Example samples from the NWPU-300 and TextRS-300 datasets. In the questions, text from the caption description is highlighted in green, and text from external knowledge is highlighted in orange.
  • Figure 4: Two generated samples from the KRSVQG model on the NWPU-300 dataset. The corresponding answer for $\widehat{Q}$ is marked in bold.