SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant

TL;DR

SQ-LLaVA addresses the bottleneck between the vision encoder and the large language model by enabling the model to self-question about visual content. It introduces a lightweight architecture featuring a prototype extractor and LoRA-based tuning, and a novel visual self-questioning objective triggered by a dedicated [vusr] token. Through a two-stage training scheme, Stage1 pre-trains vision-language alignment while Stage2 fine-tunes with instruction data, yielding state-of-the-art or competitive results across ten vision-language benchmarks and image-captioning tasks, with notable reductions in object hallucination. The findings demonstrate that leveraging self-generated questions and structured visual prototypes can enhance cross-modal understanding with fewer parameters and data, offering a practical path to more efficient and capable large vision-language models.

Abstract

Recent advances in vision-language models have shown notable generalization in broad tasks through visual instruction tuning. However, bridging the gap between the pre-trained vision encoder and the large language models (LLMs) becomes the whole network's bottleneck. To improve cross-modality alignment, existing works usually consider more visual instruction data covering a broader range of vision tasks to fine-tune the model for question-answering, which, however, is costly to obtain and has not thoroughly explored the rich contextual information contained in images. This paper first attempts to harness the overlooked context within visual instruction data, training the model to self-supervised "learning" how to ask high-quality questions. In this way, we introduce a novel framework named SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant. SQ-LLaVA exhibits proficiency in generating flexible and meaningful image-related questions while analyzing the visual clue and prior language knowledge, signifying an advanced level of generalized visual understanding. Moreover, fine-tuning SQ-LLaVA on higher-quality instruction data shows a performance improvement compared with traditional visual-instruction tuning methods. This improvement highlights the efficacy of self-questioning techniques in achieving a deeper and more nuanced comprehension of visual content across various contexts.

Figures (6)

• Figure 1: (a) Comparision between visual instruction tuning and visual self-questioning (ours) for vision-language assistant. (b) The proposed SQ-LLaVA achieves state-of-the-art performance on 9 out of 10 tasks compared with other open-ended models.
• Figure 2: Left: Questions spread more samples around a higher mean CLIPScore than answers. Right: Example of highly image-relevant questions within the visual instructional dataset for training a Vision-Language assistant.
• Figure 3: Model architecture of SQ-LLaVA. We propose prototype extractor to extract visual clustering information to enhance the visual embedding encoded by the visual encoder. SQ-LLaVA defines a new token [vusr] as specific instruction for LLM to perform visual self-questioning. Besides question answering, SQ-LLaVA treats questioning as another training objective.
• Figure 4: The input sequence used to train SQ-LLaVA. Our model is trained to predict question, answer, and where to stop. We use tokens to represent learnable tokens, where $X_q$ is the question, $X_a$ is the answer, and $<o^{d}>$ is the delimiter token. In SQ-LLaVA, the System-message = "The assistant gives helpful, detailed, and polite answers to the user's questions. Also, the assistant is a curious virtual user can ask complex questions that are relevant to the content in the image."
• Figure 5: Visual self-questioning of SQ-LLaVA-7B. Comparing to the question data provided by GPT4-V (data collected by LLaVA-v1.5 llava1-5), SQ-LLaVA can generate questions with higher diversity i.e. multiple choice and tricky questions.