Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AIDE: Agentically Improve Visual Language Model with Domain Experts

Ming-Chang Chiu, Fuxiao Liu, Karan Sapra, Andrew Tao, Yaser Jacoob, Xuezhe Ma, Zhiding Yu, Guilin Liu

TL;DR

The paper tackles the bottleneck of improving state-of-the-art Visual Language Models in the absence of superior models by introducing AIDE, a framework where VLMs autonomously improve through domain-expert collaboration. It employs two agents, a Selector and a Synthesizer, executing a four-stage workflow that identifies refinement targets, solicits expert outputs, synthesizes them with existing data, and integrates enriched samples into training. Empirical results across MMMU, MMBench, MME, MathVista, and ChartQA demonstrate notable gains using lightweight tools like PaddleOCR and Grounded-SAM, without requiring larger models or human supervision. This approach offers a scalable, resource-efficient path for continuous VLM enhancement, particularly valuable when access to larger models is restricted or unavailable.

Abstract

The enhancement of Visual Language Models (VLMs) has traditionally relied on knowledge distillation from larger, more capable models. This dependence creates a fundamental bottleneck for improving state-of-the-art systems, particularly when no superior models exist. We introduce AIDE (Agentic Improvement through Domain Experts), a novel framework that enables VLMs to autonomously enhance their capabilities by leveraging specialized domain expert models. AIDE operates through a four-stage process: (1) identifying instances for refinement, (2) engaging domain experts for targeted analysis, (3) synthesizing expert outputs with existing data, and (4) integrating enhanced instances into the training pipeline. Experiments on multiple benchmarks, including MMMU, MME, MMBench, etc., demonstrate AIDE's ability to achieve notable performance gains without relying on larger VLMs nor human supervision. Our framework provides a scalable, resource-efficient approach to continuous VLM improvement, addressing critical limitations in current methodologies, particularly valuable when larger models are unavailable to access.

AIDE: Agentically Improve Visual Language Model with Domain Experts

TL;DR

The paper tackles the bottleneck of improving state-of-the-art Visual Language Models in the absence of superior models by introducing AIDE, a framework where VLMs autonomously improve through domain-expert collaboration. It employs two agents, a Selector and a Synthesizer, executing a four-stage workflow that identifies refinement targets, solicits expert outputs, synthesizes them with existing data, and integrates enriched samples into training. Empirical results across MMMU, MMBench, MME, MathVista, and ChartQA demonstrate notable gains using lightweight tools like PaddleOCR and Grounded-SAM, without requiring larger models or human supervision. This approach offers a scalable, resource-efficient path for continuous VLM enhancement, particularly valuable when access to larger models is restricted or unavailable.

Abstract

The enhancement of Visual Language Models (VLMs) has traditionally relied on knowledge distillation from larger, more capable models. This dependence creates a fundamental bottleneck for improving state-of-the-art systems, particularly when no superior models exist. We introduce AIDE (Agentic Improvement through Domain Experts), a novel framework that enables VLMs to autonomously enhance their capabilities by leveraging specialized domain expert models. AIDE operates through a four-stage process: (1) identifying instances for refinement, (2) engaging domain experts for targeted analysis, (3) synthesizing expert outputs with existing data, and (4) integrating enhanced instances into the training pipeline. Experiments on multiple benchmarks, including MMMU, MME, MMBench, etc., demonstrate AIDE's ability to achieve notable performance gains without relying on larger VLMs nor human supervision. Our framework provides a scalable, resource-efficient approach to continuous VLM improvement, addressing critical limitations in current methodologies, particularly valuable when larger models are unavailable to access.

Paper Structure

This paper contains 22 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation.
  4. Specialized Models.
  5. Data Synthesis and Augmentation.
  6. Aide Framework
  7. Selector.
  8. Synthesizer.
  9. Integration
  10. Experiment
  11. Setup.
  12. Expert Tool Choice.
  13. Integration.
  14. Results.
  15. Ablations
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Aide Workflow. Aide consists of two agents, a Selector and a Synthesizer. The Selector interacts with the data instances and autonomously invoke the expert tools as it deems fit. The Synthesizer collects information from the original data instances along with outputs from the select experts and generate enriched response.
  • Figure 2: Small-step Prompting. We observe even when VLM is able to answer the query (middle-column), sometimes the instruction following is not stable. And simplifying the prompt into smaller steps by giving the answer (last column) gives more detailed responses.
  • Figure 3: Left: Breakdown of selected data instances by VLM-Selector. Synthdog takes the most proportion of the selection. Right: Ratio of data instances selected by the VLM-Selector to the total instances in the original Cambrian-1.
  • Figure 4: Comparisons of the original and the new answer produced by Aide. Our Aide workflow enriches the responses.