ConVQG: Contrastive Visual Question Generation with Multimodal Guidance

TL;DR

ConVQG tackles the challenge of generating image-grounded questions that also adhere to textual constraints by introducing dual modality–specific contrastive objectives. The method extends a BLIP-based vision-language framework with two auxiliary question-generators (IQGM and TQGM) and a joint contrastive loss that pushes multimodal questions toward the ground-truth and away from unimodal baselines, yielding image-grounded, text-guided, and knowledge-rich questions. Empirical results across knowledge-aware and standard VQG benchmarks show consistent improvements over state-of-the-art methods, with robust ablations and favorable human evaluations. The approach also demonstrates transferability to unseen datasets (e.g., FVQA) and offers flexible constraints (answers, knowledge triplets, captions) for diverse VQG applications in visual dialog systems.

Abstract

Asking questions about visual environments is a crucial way for intelligent agents to understand rich multi-faceted scenes, raising the importance of Visual Question Generation (VQG) systems. Apart from being grounded to the image, existing VQG systems can use textual constraints, such as expected answers or knowledge triplets, to generate focused questions. These constraints allow VQG systems to specify the question content or leverage external commonsense knowledge that can not be obtained from the image content only. However, generating focused questions using textual constraints while enforcing a high relevance to the image content remains a challenge, as VQG systems often ignore one or both forms of grounding. In this work, we propose Contrastive Visual Question Generation (ConVQG), a method using a dual contrastive objective to discriminate questions generated using both modalities from those based on a single one. Experiments on both knowledge-aware and standard VQG benchmarks demonstrate that ConVQG outperforms the state-of-the-art methods and generates image-grounded, text-guided, and knowledge-rich questions. Our human evaluation results also show preference for ConVQG questions compared to non-contrastive baselines.

Figures (11)

• Figure 1: ConVQG at a glance. An image and a text input are processed through a multimodal module , leading to the embedding $Q_{it}$. Pre-trained modules (detailed in Fig. \ref{['fig:framework']}) produce image-only and text-only question embeddings ($Q_i$ and $Q_t$). A contrastive loss is then optimized to make $Q_{it}$ close to the real question embedding $Q_{gt}$ and far from the single modality ones. By design, ConVQG generates questions that are image-grounded (in green) and that meet the requirements of the text constraint (in yellow).
• Figure 2: Pipeline of the ConVQG method. During training, an encoder-decoder VQG framework is powered by two additional branches for image-based question generation (IQGM) and text-based question generation (TQGM) (left part, the locker icon means the model is frozen). Then, contrastive losses discriminate image-text joint embeddings with the one from single modality only (right part). During inference, only the encoder-decoder framework is activated.
• Figure 3: Examples from K-VQG dataset with knowledge triplets as inputs. In the text, green color denotes the sequence that is related to image content, while yellow color denotes the information related to the text input. Red color indicates wrong expressions, not related to the image nor the text input. Note: the raw input/output of the model is reported, without correcting grammar or syntax errors made by the generative model.
• Figure 4: Question generation by ConVQG. Given the same image, it can generate different text-guided questions. Given the same text input, it can generate image-specific questions.
• Figure 5: Transfer results on the FVQA dataset.