LOVA3: Learning to Visual Question Answering, Asking and Assessment

TL;DR

LOVA$^3$ addresses the gap in multimodal large language models by endowing them with two human-like abilities: asking and assessing. It introduces GenQA to generate diverse question–answer pairs and EvalQA to judge answer correctness with constructive feedback, enabled by the EvalQABench benchmark. Built on a LLaVA-1.5 backbone, LOVA$^3$ integrates three training tasks and demonstrates consistent gains across ten multimodal benchmarks and the EvalQABench test, validating that asking and evaluation enrich multimodal understanding. The approach offers scalable improvements and opens avenues for richer human–AI interactions, with the provided codebase and data pipelines for reproducibility.

Abstract

Question answering, asking, and assessment are three innate human traits crucial for understanding the world and acquiring knowledge. By enhancing these capabilities, humans can more effectively utilize data, leading to better comprehension and learning outcomes. Current Multimodal Large Language Models (MLLMs) primarily focus on question answering, often neglecting the full potential of questioning and assessment skills. Inspired by the human learning mechanism, we introduce LOVA3, an innovative framework named "Learning tO Visual question Answering, Asking and Assessment," designed to equip MLLMs with these additional capabilities. Our approach involves the creation of two supplementary training tasks GenQA and EvalQA, aiming at fostering the skills of asking and assessing questions in the context of images. To develop the questioning ability, we compile a comprehensive set of multimodal foundational tasks. For assessment, we introduce a new benchmark called EvalQABench, comprising 64,000 training samples (split evenly between positive and negative samples) and 5,000 validation and testing samples. We posit that enhancing MLLMs with the capabilities to answer, ask, and assess questions will enhance their multimodal comprehension, ultimately improving overall performance. To validate this hypothesis, we train MLLMs using the LOVA3 framework and evaluate them on a range of multimodal datasets and benchmarks. Our results demonstrate consistent performance gains, underscoring the critical role of these additional tasks in fostering comprehensive intelligence in MLLMs. The code is available at https://github.com/showlab/LOVA3.

Figures (9)

• Figure 1: Comparison of three abilities reveals that LLaVA1.5 excels in providing answers but struggles in asking accurate questions and assessing question-answer pairs.
• Figure 2: Illustration of the proposed pipeline for generating negative answers and feedback.
• Figure 3: Examples from the manual filtering and error correction process. Red text indicates error answers, while Green text represents manually corrected answers.
• Figure 4: Illustration of the model training of LOVA$^3$.
• Figure 5: The word cloud of total negative answers.