Black-box Model Ensembling for Textual and Visual Question Answering via Information Fusion

TL;DR

This work tackles the challenge of improving textual and multimodal question answering when fine-tuning large black-box models is impractical. It introduces InfoSel, a data-efficient ensemble that dynamically selects the best base model without relying on prediction confidences, employing task-specific architectures (InfoSel-TT for TQA and InfoSel-MT for VQA) and optional refinements (FT and InfoSel*) to cover unseen labels. Through data-efficient training on Mini datasets derived from SQuAD-V2, NQ-Open, GQA, and VizWiz, InfoSel achieves substantive gains over baselines and base models, including up to $+5.19\%$ F1 on TQA and $+31.63\%$ on VQA with the InfoSel* variant, using as few as $1{,}000$ labeled examples. These results demonstrate effective multimodal information fusion and dynamic winner selection, offering a practical, scalable approach to enhancing performance in real-world, API-constrained settings where access to model internals and confidences is limited.

Abstract

A diverse range of large language models (LLMs), e.g., ChatGPT, and visual question answering (VQA) models, e.g., BLIP, have been developed for solving textual and visual question answering tasks. However, fine-tuning these models is either difficult, as it requires access via APIs, rendering them as black-boxes, or costly due to the need of tuning a large number of parameters. To address this, we introduce InfoSel, a data-efficient ensemble method that learns to dynamically pick the winner from existing black-box models for predictions on both textual and multimodal visual question answering tasks. Unlike traditional ensemble models, InfoSel does not rely on prediction probabilities or confidences, which typically are not available in black-box models. Experimental results on four datasets demonstrate that our approach achieves an absolute increase of up to +5.19\% in the F1-score compared to standalone LLMs using only 1K training instances.

Figures (6)

• Figure 1: Architecture of our InfoSel (step ①) , FT (step ②) and InfoSel$^*$ (step ③) models. $M^l_*$ and $M^v_*$ refer to black-box LLMs and VQA base models respectively, which are not trainable. The number of these base models is flexible, and is not restricted to 3 as in the figure. The models on the left (suffixed with -TT) are trained for the TQA tasks, while the models on the right (suffixed with -MT) are trained for the VQA tasks. All our models are trained independently. Note that FT and InfoSel$^*$ (step ② and ③) are optional and are best suited for datasets that contain a high percentage of labels that the base models have not been exposed to.
• Figure 2: Top 7 most frequent answers of VQA v2 (pre-trained dataset of VQA base models), GQA and VizWiz (task-specific datasets for ensemble training). This explains why InfoSel performs poorly in Mini-Viz, as the new label "unanswerable" is the top frequent answer in VizWiz, but this new label has been exposed to base models that are pre-trained on VQA v2.
• Figure 3: TQA test performance of InfoSel compared to baselines over increasing size of training data. Best base represented the best performance of the base models.
• Figure 4: VQA test performance of InfoSel compared to baselines over increasing size of training data. Best base represented the best performance of the base models. The FT method is outperforming all the ensemble methods (OLA, PairRanker and InfoSel) due to the underexposure of a new label "unanswerable" to the ensembled base models.
• Figure 5: The portions of answers selected from different base models by InfoSel models on Mini-SDv2 and Mini-NQ test data.