Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing

TL;DR

The proposed KRSVQG model incorporates related knowledge triplets from external knowledge sources to broaden the question content, while employing image captioning as an intermediary representation to ground questions to the corresponding images, enabling the model’s adaptation to low data regimes.

Abstract

With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing semantic image retrieval. However, current automatically generated questions tend to be simplistic and template-based, which hinders the deployment of question answering or visual dialogue systems for real-world applications. To enrich and diversify the questions with both image content and commonsense knowledge, we propose a Knowledge-aware Remote Sensing Visual Question Generation model (KRSVQG). The proposed model incorporates related knowledge triplets from external knowledge sources to broaden the question content, while employing image captioning as an intermediary representation to ground questions to the corresponding images. Moreover, KRSVQG utilizes a vision-language pre-training and fine-tuning strategy, enabling the model's adaptation to low data regimes. To evaluate the proposed KRSVQG model, we construct two knowledge-aware remote sensing visual question generation datasets: the NWPU-300 dataset and the TextRS-300 dataset. Evaluations, including metrics and human assessment, demonstrate that KRSVQG outperforms existing methods and leads to rich questions, grounded in both image and domain knowledge. As a key practice in vision-language research, knowledge-aware visual question generation advances the understanding of image content beyond pixels, facilitating the development of knowledge-enriched vision-language systems with vision-grounded human commonsense.

Figures (8)

• Figure 1: An example of image-based questions and a knowledge-aware question. In the questions, text related to image description is highlighted in green, while text related to external knowledge is in orange.
• Figure 2: The annotation process for the NWPU-300 and TextRS-300 datasets involves three stages: First, Triplet Retrieval, where the knowledge triplets related to the original caption are retrieved from ConceptNet. Next, Triplet Ranking, where the retrieved triplets are ranked based on their similarity to the caption. Finally, Question Annotation, where we annotate a question based on the selected triplet.
• Figure 3: The distribution of question lengths in the TextRS-300 dataset and TextRS-VQA dataset.
• Figure 4: The examples from TextRS-VQA dataset and TextRS-300 dataset. In the questions, text from the caption description is highlighted in green, and text from external knowledge is highlighted in orange.
• Figure 5: The KRSVQG model and its four components (See Section \ref{['sec:arch']}): (i) image encoder ($\mathbf{ViT}$); (ii) caption decoder ($\mathbf{BERT_{CapDec}}$); (iii) text encoder ($\mathbf{BERT_{TextEnc}}$); and (iv) question decoder ($\mathbf{BERT_{QueDec}}$). With an image ($I$) and a knowledge sentence ($S$) as inputs, the model generates a knowledge-aware question ($\widehat{Q}$) and caption ($\widehat{C}$) based on the image ($I$) and the knowledge sentence ($S$). KRSVQG is trained under the proposed training strategy (See Section \ref{['sec:training']}): In vision pre-training (VPT), tunable parameters are confined to the vision module (upper row). During language pre-training (LPT), both vision and language modules are jointly trained. In the fine-tuning (FT) stage, the vision module initializes from VPT weights, while the language module initializes from LPT weights. Both modules are updated during FT.$\oplus$ means addition and normalization. Dotted arrows indicate the inference path.