Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Multi-Image Question Answering via Submodular Subset Selection

Aaryan Sharma, Shivansh Gupta, Samar Agarwal, Vishak Prasad C., Ganesh Ramakrishnan

TL;DR

Multi-image question answering (MIQA) systems struggle with scalability and retrieval accuracy when reasoning over large image collections. The authors propose a submodular subset selection layer that uses anchor/reference images and query-aware objectives (GraphCut, Facility Location, LogDet) to pre-filter haystacks before feeding them into the MIRAGE retriever, enabling more scalable retrieval. Key findings show anchor-based subset selection improves ground-truth inclusion and retriever success, with GraphCut Mutual Information often outperforming mixed-function strategies, and data augmentation providing additional gains. The approach enhances scalability and efficiency for MIQA on large haystacks, offering a practical path toward robust retrieval-augmented vision-language systems.

Abstract

Large multimodal models (LMMs) have achieved high performance in vision-language tasks involving single image but they struggle when presented with a collection of multiple images (Multiple Image Question Answering scenario). These tasks, which involve reasoning over large number of images, present issues in scalability (with increasing number of images) and retrieval performance. In this work, we propose an enhancement for retriever framework introduced in MIRAGE model using submodular subset selection techniques. Our method leverages query-aware submodular functions, such as GraphCut, to pre-select a subset of semantically relevant images before main retrieval component. We demonstrate that using anchor-based queries and augmenting the data improves submodular-retriever pipeline effectiveness, particularly in large haystack sizes.

Enhancing Multi-Image Question Answering via Submodular Subset Selection

TL;DR

Multi-image question answering (MIQA) systems struggle with scalability and retrieval accuracy when reasoning over large image collections. The authors propose a submodular subset selection layer that uses anchor/reference images and query-aware objectives (GraphCut, Facility Location, LogDet) to pre-filter haystacks before feeding them into the MIRAGE retriever, enabling more scalable retrieval. Key findings show anchor-based subset selection improves ground-truth inclusion and retriever success, with GraphCut Mutual Information often outperforming mixed-function strategies, and data augmentation providing additional gains. The approach enhances scalability and efficiency for MIQA on large haystacks, offering a practical path toward robust retrieval-augmented vision-language systems.

Abstract

Large multimodal models (LMMs) have achieved high performance in vision-language tasks involving single image but they struggle when presented with a collection of multiple images (Multiple Image Question Answering scenario). These tasks, which involve reasoning over large number of images, present issues in scalability (with increasing number of images) and retrieval performance. In this work, we propose an enhancement for retriever framework introduced in MIRAGE model using submodular subset selection techniques. Our method leverages query-aware submodular functions, such as GraphCut, to pre-select a subset of semantically relevant images before main retrieval component. We demonstrate that using anchor-based queries and augmenting the data improves submodular-retriever pipeline effectiveness, particularly in large haystack sizes.
Paper Structure (9 sections, 8 figures)

This paper contains 9 sections, 8 figures.

Table of Contents

  1. Problem Statement
  2. Related Works
  3. Approach of Solving
  4. Subset Selection using Submodular Functions
  5. Integration with Mirage Framework
  6. Experiments and Results
  7. Subset Selection
  8. Analyzing the Retriever
  9. Future Works

Figures (8)

  • Figure 1: Evaluation of MIRAGE and Retriever
  • Figure 2: Pipeline for subset selection
  • Figure 3: Retriever performance using Target-based subset selection. The x-axis represents the subset size as a percentage of the total haystack. Different legends are for different size of haystacks.
  • Figure 4: Performance comparison of single-function (GCMI) vs. mixed-function (GCMI, FLVMI, LogDet) subset selection for haystack sizes of 500 and 1000.
  • Figure 5: Comparison of subset selection using Anchor vs. Target images as queries.
  • ...and 3 more figures