Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent VQA: Exploring Multi-Agent Foundation Models in Zero-Shot Visual Question Answering

Bowen Jiang, Zhijun Zhuang, Shreyas S. Shivakumar, Dan Roth, Camillo J. Taylor

TL;DR

This work proposes an adaptive multi-agent system, named Multi-Agent VQA, to overcome the limitations of foundation models in object detection and counting by using specialized agents as tools by making it more practical and robust in the open world.

Abstract

This work explores the zero-shot capabilities of foundation models in Visual Question Answering (VQA) tasks. We propose an adaptive multi-agent system, named Multi-Agent VQA, to overcome the limitations of foundation models in object detection and counting by using specialized agents as tools. Unlike existing approaches, our study focuses on the system's performance without fine-tuning it on specific VQA datasets, making it more practical and robust in the open world. We present preliminary experimental results under zero-shot scenarios and highlight some failure cases, offering new directions for future research.

Multi-Agent VQA: Exploring Multi-Agent Foundation Models in Zero-Shot Visual Question Answering

TL;DR

This work proposes an adaptive multi-agent system, named Multi-Agent VQA, to overcome the limitations of foundation models in object detection and counting by using specialized agents as tools by making it more practical and robust in the open world.

Abstract

This work explores the zero-shot capabilities of foundation models in Visual Question Answering (VQA) tasks. We propose an adaptive multi-agent system, named Multi-Agent VQA, to overcome the limitations of foundation models in object detection and counting by using specialized agents as tools. Unlike existing approaches, our study focuses on the system's performance without fine-tuning it on specific VQA datasets, making it more practical and robust in the open world. We present preliminary experimental results under zero-shot scenarios and highlight some failure cases, offering new directions for future research.
Paper Structure (16 sections, 2 figures, 3 tables)

This paper contains 16 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Initial attempt
  4. Reattempt by adaptively calling multi-agents
  5. LLM parsing agent
  6. Object detection agent
  7. LVLM object description agent
  8. LVLM reattempts the question
  9. Object counting agent
  10. LLM answer grading agent
  11. Experiments
  12. Datasets
  13. Results and Zero-Shot VQA benchmark
  14. Ablation study
  15. Limitations and failure examples
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Overview of the adaptive Multi-Agent VQA System. The process begins with an LVLM attempting to answer a visual question directly. An LLM parsing agent automatically detects challenging cases and calls specialized agents, including object-detection and counting models as tools. The LVLM would reattempt the question, with the response assessed by LLM-based graders for a majority vote.
  • Figure 2: An example of the failure case.