Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?

Sazia Tabasum Mim, Jack Morris, Manish Dhakal, Yanming Xiu, Maria Gorlatova, Yi Ding

TL;DR

The paper tackles whether a unimodal, text-only LLM can autonomously generate preference feedback to tune a multimodal vision-language model (VLM). By generating multiple VLM captions, using an LLM agent to rank them, and applying Direct Preference Optimization, the VLM is fine-tuned to produce task-aligned, concise, and visually faithful descriptions. Experiments on MUStARD (sarcasm) and UR-FUNNY (humor) across six LLM agents show consistent performance gains over baselines, with a notable 64.6% human-alignment rate with the LLM’s preferences. The approach demonstrates a scalable, interpretable pathway to augment existing LLMs with multimodal reasoning without full multimodal retraining, though it notes limitations like offline optimization and occasional hallucinations, pointing to online adaptation and hazard mitigation as future directions.

Abstract

To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a multimodal model? To answer this, we propose a method that enables a language agent to give feedback to a vision-language model (VLM) to adapt text generation to the agent's preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can generate multimodal scene descriptions to help the LLM better understand multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference alignment rate between the LLM's choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.

Can a Unimodal Language Agent Provide Preferences to Tune a Multimodal Vision-Language Model?

TL;DR

The paper tackles whether a unimodal, text-only LLM can autonomously generate preference feedback to tune a multimodal vision-language model (VLM). By generating multiple VLM captions, using an LLM agent to rank them, and applying Direct Preference Optimization, the VLM is fine-tuned to produce task-aligned, concise, and visually faithful descriptions. Experiments on MUStARD (sarcasm) and UR-FUNNY (humor) across six LLM agents show consistent performance gains over baselines, with a notable 64.6% human-alignment rate with the LLM’s preferences. The approach demonstrates a scalable, interpretable pathway to augment existing LLMs with multimodal reasoning without full multimodal retraining, though it notes limitations like offline optimization and occasional hallucinations, pointing to online adaptation and hazard mitigation as future directions.

Abstract

To explore a more scalable path for adding multimodal capabilities to existing LLMs, this paper addresses a fundamental question: Can a unimodal LLM, relying solely on text, reason about its own informational needs and provide effective feedback to optimize a multimodal model? To answer this, we propose a method that enables a language agent to give feedback to a vision-language model (VLM) to adapt text generation to the agent's preferences. Our results from different experiments affirm this hypothesis, showing that LLM preference feedback significantly enhances VLM descriptions. Using our proposed method, we find that the VLM can generate multimodal scene descriptions to help the LLM better understand multimodal context, leading to improvements of maximum 13% in absolute accuracy compared to the baseline multimodal approach. Furthermore, a human study validated our AI-driven feedback, showing a 64.6% preference alignment rate between the LLM's choices and human judgments. Extensive experiments provide insights on how and why the method works and its limitations.
Paper Structure (40 sections, 2 equations, 2 figures, 9 tables, 1 algorithm)

This paper contains 40 sections, 2 equations, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning from AI Feedback (RLAIF)
  4. LLM-Based Multimodal Reasoning
  5. Preference Optimization in Model Alignment
  6. Methodology
  7. Generating Captions with a VLM
  8. Generating Agent Preferences
  9. Agent preferences with ground truth knowledge (With GT)
  10. Agent preferences without ground truth knowledge (Without GT)
  11. Agent preferences with in-context examples (With ICL)
  12. Sorting preferences
  13. Preference Optimization
  14. Agent Evaluation Process
  15. Experimental Setup
  16. ...and 25 more sections

Figures (2)

  • Figure 1: An overview of the training pipeline. (a) First, a VLM generates diverse descriptions with a set of prompts; Second, (b) descriptions are ranked using an LLM Agent to generate preference dataset; Lastly, (c) generated preference dataset is used for DPO-based optimization to fine tune the VLM. The agent model stays fixed.
  • Figure 2: Overview of the evaluation process. In the unimodal evaluation pipeline shown in (a), only the conversation context and the final statement is given to the Agent to determine the label. (b) demonstrates how multimodal information is incorporated during the evaluation process. The VLM acts like a feature extractor by converting visual features to text descriptions. This is fed to the LLM. (c) shows how the preference tuned VLM (Optimized Vision Language Model) is incorporated into the pipeline.