Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQA

TL;DR

The paper tackles zero-shot knowledge-based VQA by showing that a single knowledge-generation strategy is often insufficient. It introduces DietCoke, a three-stage framework (Diversification, Rationalization, Ensemble) that creates three decision contexts—caption-only, short-form knowledge, and long-form knowledge—and generates two types of rationales (automatic and mechanistic) to enable an LLM-based ensemble to select the final answer. Empirical results on OK-VQA and A-OKVQA demonstrate state-of-the-art zero-shot performance and reveal strong synergy among diverse QA strategies and rationales. The approach highlights the value of decorrelated reasoning strategies and rationale-guided fusion for scalable, inference-time adaptable K-VQA, while acknowledging compute-time trade-offs and biases inherent in LLMs.

Abstract

Knowledge-based Visual Question-answering (K-VQA) often requires the use of background knowledge beyond the image. However, we discover that a single knowledge generation strategy is often insufficient for all K-VQA questions. To this end, we propose Diversification, Evidence Truncation, and Combination for Knowledge-based Elucidation (DietCoke), which utilizes a bundle of complementary question-answering tactics and aggregates their answers using textual rationales. DietCoke comprises of three stages: diversification, rationalization, and ensemble. The diversification stage generates three distinctive decision contexts, each leading to its own answer candidate. The rationalization stage generates two rationales, the automatic rationale and the mechanistic rationale, for each answer candidate using decorrelated techniques. Finally, in the ensemble stage, an LLM informed by the rationales selects one answer from the three candidates. Experiments show that DietCoke significantly outperforms state-of-the-art LLM-based baselines by 2.8% on OK-VOA and 4.7% on A-OKVOA and that the strategies in the ensembles are highly complementary. Code is available at: https://github.com/limiaoyu/DietCoke

Figures (8)

• Figure 1: The three K-VQA questions are best answered using distinctive decision contexts, including image captions, (a), and two forms of generated knowledge statements, (b) and (c). This is due to (1) the difficulty in controlling the generation of captions and knowledge statements and ensuring they contain relevant information, and (2) the inability of LLMs to identify correct answers from noisy contexts.
• Figure 2: The framework of DietCoke. The diversification phase contains three question-answering strategies, which generate three different decision contexts for the answer-predicting LLM. The short-form knowledge contains a single sentence, whereas the long-form knowledge consists of a paragraph of background information. The caption-only strategy generates only image captions and no background knowledge. For each strategy, the LLM generates an answer candidate. In the rationalization phase, for each candidate, we generate an Automatic (A.) Rationale and a Mechanistic (M.) Rationale, which guide the ensemble LLM in selecting the best answer.
• Figure 3: The heatmap of relevance scores of sentences in captions. The relevance score represents the contribution of the sentence to the answer.
• Figure 4: The success rate of the other two strategies on all questions successfully answered by one strategy. In left group, we show the percentage of correctly answered questions by SK and LK out of all the questions that are correctly answered by CO. The other two bar groups are drawn similarly.
• Figure 5: Conditional probabilities of the three answer candidates being selected conditioned on the exact occurrence of the answer in the rationales. There are six different conditions, including the answer candidate occurring in the automatic rationale (AR), the mechanistic rationale (MR), either rationale, and their respective negations. The abbreviations of strategy names are: CO = caption-only, SK = short-form knowledge, and LK = long-form knowledge.