Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks

Jusung Lee, Sungguk Cha, Younghyun Lee, Cheoljong Yang

TL;DR

The paper tackles extending multimodal LLMs to domain-specific visual tasks while preserving vision-language capabilities. It introduces Visual Question Answering Instruction (VQA-IN), a data transformation framework that unifies domain-specific visual data and vision-language datasets into a coherent question-answer format, enabling multitask learning with sLLMs. Through experiments across BLIP-2, InstructBLIP, and OpenFlamingo architectures, VQA-IN demonstrates improvements on standard vision-language benchmarks and domain-specific tasks, highlighting the method’s effectiveness regardless of model size. The work suggests a practical path toward robust multimodal assistants capable of handling diverse visual reasoning tasks without relying on ever-larger LLMs.

Abstract

Having revolutionized natural language processing (NLP) applications, large language models (LLMs) are expanding into the realm of multimodal inputs. Owing to their ability to interpret images, multimodal LLMs (MLLMs) have been primarily used for vision-language tasks. Currently, MLLMs have not yet been extended for domain-specific visual tasks, which require a more explicit understanding of visual information. We developed a method to transform domain-specific visual and vision-language datasets into a unified question answering format called Visual Question Answering Instruction (VQA-IN), thereby extending MLLM to domain-specific tasks. The VQA-IN was applied to train multiple MLLM architectures using smaller versions of LLMs (sLLMs). The experimental results indicated that the proposed method achieved a high score metric on domainspecific visual tasks while also maintaining its performance on vision-language tasks in a multitask manner.

Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks

TL;DR

The paper tackles extending multimodal LLMs to domain-specific visual tasks while preserving vision-language capabilities. It introduces Visual Question Answering Instruction (VQA-IN), a data transformation framework that unifies domain-specific visual data and vision-language datasets into a coherent question-answer format, enabling multitask learning with sLLMs. Through experiments across BLIP-2, InstructBLIP, and OpenFlamingo architectures, VQA-IN demonstrates improvements on standard vision-language benchmarks and domain-specific tasks, highlighting the method’s effectiveness regardless of model size. The work suggests a practical path toward robust multimodal assistants capable of handling diverse visual reasoning tasks without relying on ever-larger LLMs.

Abstract

Having revolutionized natural language processing (NLP) applications, large language models (LLMs) are expanding into the realm of multimodal inputs. Owing to their ability to interpret images, multimodal LLMs (MLLMs) have been primarily used for vision-language tasks. Currently, MLLMs have not yet been extended for domain-specific visual tasks, which require a more explicit understanding of visual information. We developed a method to transform domain-specific visual and vision-language datasets into a unified question answering format called Visual Question Answering Instruction (VQA-IN), thereby extending MLLM to domain-specific tasks. The VQA-IN was applied to train multiple MLLM architectures using smaller versions of LLMs (sLLMs). The experimental results indicated that the proposed method achieved a high score metric on domainspecific visual tasks while also maintaining its performance on vision-language tasks in a multitask manner.
Paper Structure (14 sections, 2 figures, 2 tables)

This paper contains 14 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Related Work
  4. Multi-modal Large Language Model
  5. Instruction Tuning
  6. Method
  7. VQA-IN for Vision-language Task
  8. VQA-IN for Domain-specific Visual Task
  9. Sentence Length Control
  10. VQA-IN for MLLM Architecture
  11. Experiments
  12. Evaluation on Vision-language Tasks
  13. Evaluation on Domain-specific Visual Tasks
  14. Conclusions

Figures (2)

  • Figure 1: Examples of the proposed visual question answering instruction (VQA-IN), which unlocks MLLMs to enable the creation of a domain-specific visual multitask model.
  • Figure 2: Examples of VQA-IN transformation. The first row displays the original dataset format for each task, while the second row shows the results of transforming these diverse formats into a unified question answering format.