Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning
Yuqi Pang, Bowen Yang, Haoqin Tu, Yun Cao, Zeyu Zhang
TL;DR
The paper addresses the challenge of multimodal reasoning with large language models by proposing MVCD, a plug-and-play framework that adds a visual perception module and uses contrastive-example decoding (CED) with in-context learning to guide decoding without task-specific training. Visual signals are converted to text via modules for images ($N$ CLIP-derived tags/attributes and $N$ BLIP captions) and videos ($N$ tags/attributes via LanguageBind and $N$ captions via Video-LLaVA), forming descriptive features $F$ and context examples $E^i=(F^i,T^i)$. The method contrasts the base token distribution $p(t_j)$ with the adjusted $ ilde{p}(t_j)$ using adaptive constraints to focus on example-derived information, with key quantities such as $V_{ ext{head}}$ and $ ext{CED ext{-}score}(t_j)= ext{log} rac{ ilde{p}(t_j)}{p(t_j)}$. Experiments on five VQA datasets show MVCD improves accuracy across different LLMs and settings without additional training, demonstrating enhanced visual perception and robust multimodal reasoning. The work highlights a practical path to leverage pretrained visual encoders and LLMs for multimodal tasks and suggests future comparisons across alternative pretrained backbones.
Abstract
Although Large Language Models (LLMs) excel in reasoning and generation for language tasks, they are not specifically designed for multimodal challenges. Training Multimodal Large Language Models (MLLMs), however, is resource-intensive and constrained by various training limitations. In this paper, we propose the Modular-based Visual Contrastive Decoding (MVCD) framework to move this obstacle. Our framework leverages LLMs' In-Context Learning (ICL) capability and the proposed visual contrastive-example decoding (CED), specifically tailored for this framework, without requiring any additional training. By converting visual signals into text and focusing on contrastive output distributions during decoding, we can highlight the new information introduced by contextual examples, explore their connections, and avoid over-reliance on prior encoded knowledge. MVCD enhances LLMs' visual perception to make it see and reason over the input visuals. To demonstrate MVCD's effectiveness, we conduct experiments with four LLMs across five question answering datasets. Our results not only show consistent improvement in model accuracy but well explain the effective components inside our decoding strategy. Our code will be available at https://github.com/Pbhgit/MVCD.