Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy

TL;DR

Proposed Visual-Modality Token Compression and Cross-Modality Attention Inhibition strategies are proposed to alleviate the gap between MLLMs and LLMs by inhibiting the influence of irrelevant visual tokens during content generation, increasing the instruction-following ability of the MLLMs while retaining their multimodal understanding capacity.

Abstract

Large Language Models (LLMs) have strong instruction-following capability to interpret and execute tasks as directed by human commands. Multimodal Large Language Models (MLLMs) have inferior instruction-following ability compared to LLMs. However, there is a significant gap in the instruction-following capabilities between the MLLMs and LLMs. In this study, we conduct a pilot experiment, which demonstrates that spatially down-sampling visual tokens significantly enhances the instruction-following capability of MLLMs. This is attributed to the substantial redundancy in visual modality. However, this intuitive method severely impairs the MLLM's multimodal understanding capability. In this paper, we propose Visual-Modality Token Compression (VMTC) and Cross-Modality Attention Inhibition (CMAI) strategies to alleviate this gap between MLLMs and LLMs by inhibiting the influence of irrelevant visual tokens during content generation, increasing the instruction-following ability of the MLLMs while retaining their multimodal understanding capacity. In VMTC module, the primary tokens are retained and the redundant tokens are condensed by token clustering and merging. In CMAI process, we aggregate text-to-image attentions by text-to-text attentions to obtain a text-to-image focus score. Attention inhibition is performed on the text-image token pairs with low scores. Our comprehensive experiments over instruction-following capabilities and VQA-V2, GQA, TextVQA, MME and MMBench five benchmarks, demonstrate that proposed strategy significantly enhances the instruction following capability of MLLMs while preserving the ability to understand and process multimodal inputs.

Figures (4)

• Figure 1: Illustration of the gap in the instruction-following ability between GPT4 achiam2023gpt and GPT-4V gpt4v. Under the same instruction, GPT-4V cannot generate outputs that meet the requirements of the instruction. However, when the image in the input is replaced with a detailed description of the image generated by GPT-4, the model can correctly follow the provided instructions.
• Figure 2: Illustration of instruction following performance and multimodal understanding capability of LLaVA-1.5 using different spatial down-sampling ratio. In Figure \ref{['fig:json']} and \ref{['fig:keywords']}, the instruction-following performances are significantly improved as the down-sampling ratio increases. In Figure \ref{['fig:gqa']} and \ref{['fig:mme']}, down-sampling operations significantly affect the model's multimodal understanding capability.
• Figure 3: Overview of our proposed framework. Our proposed framework incorporates two principal components: the VMTC module and the CMAI module. The VMTC module is strategically positioned within several transformer blocks of the visual encoder, specifically between the self-attention and feed-forward networks. This module aims to compress image redundancy by retaining primary tokens while clustering and merging redundant tokens. Meanwhile, the CMAI module is integrated into the self-attention modules of transformer blocks in LLMs, effectively inhibiting the influence of redundant image tokens.
• Figure 4: Qualitative results of instruction following capabilities, comparing our proposed method with InstructBlip dai2023instructblip, IDEFICS idefics, Qwen-VL bai2023qwen, MiniGPT-v2 chen2023minigpt, SPHINX lin2023sphinx, mPLUG-Owl2 yu2024rlhf and LLaVA-1.5 liu2024improved.