COCO is "ALL'' You Need for Visual Instruction Fine-tuning

TL;DR

The paper investigates why current visual instruction fine-tuning (IFT) datasets lead to degraded open-ended response quality in multi-round dialog. It argues that high-quality, diverse COCO-based instruction annotations can address this bias and proposes a COCO-centric IFT dataset created by merging COCO and Visual Genome annotations and applying rule-based templates. Fine-tuning end-to-end MLLMs on this dataset yields improved performance on open-ended benchmarks in both single-round and multi-round contexts, and shows comparable results to baselines on MM-Vet and InfiMM-Eval where the original model deteriorates under multi-round evaluation. The work concludes that COCO alone can be a sufficient source for visual IFT and calls for more nuanced benchmarks beyond traditional caption/VQA metrics.

Abstract

Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. Visual instruction fine-tuning (IFT) is a vital process for aligning MLLMs' output with user's intentions. High-quality and diversified instruction following data is the key to this fine-tuning process. Recent studies propose to construct visual IFT datasets through a multifaceted approach: transforming existing datasets with rule-based templates, employing GPT-4 for rewriting annotations, and utilizing GPT-4V for visual dataset pseudo-labeling. LLaVA-1.5 adopted similar approach and construct LLaVA-mix-665k, which is one of the simplest, most widely used, yet most effective IFT datasets today. Notably, when properly fine-tuned with this dataset, MLLMs can achieve state-of-the-art performance on several benchmarks. However, we noticed that models trained with this dataset often struggle to follow user instructions properly in multi-round dialog. In addition, tradition caption and VQA evaluation benchmarks, with their closed-form evaluation structure, are not fully equipped to assess the capabilities of modern open-ended generative MLLMs. This problem is not unique to the LLaVA-mix-665k dataset, but may be a potential issue in all IFT datasets constructed from image captioning or VQA sources, though the extent of this issue may vary. We argue that datasets with diverse and high-quality detailed instruction following annotations are essential and adequate for MLLMs IFT. In this work, we establish a new IFT dataset, with images sourced from the COCO dataset along with more diverse instructions. Our experiments show that when fine-tuned with out proposed dataset, MLLMs achieve better performance on open-ended evaluation benchmarks in both single-round and multi-round dialog setting.

Figures (3)

• Figure 1: Demonstration of different models' responses under multi-round dialog setting.
• Figure 2: QA and Caption datasets derived from COCO and Visual Genome.
• Figure 3: Examples of annotations from caption and VQA datasets.