Probing the Limits of Stylistic Alignment in Vision-Language Models

TL;DR

This work investigates data-efficient alignment of small vision–language models to subjective caption styles (humor and romance) using preference-based objectives, notably SimPO, alongside zero-shot prompting and supervised fine-tuning. It evaluates on two datasets (New Yorker captions and FlickrStyle10K) with explicit data budgets, reporting win-rate and style-classification accuracy to trace data-efficiency curves. The study introduces a reproducible protocol and finds that peak performance can be reached with as little as ~10% of available preference data, indicating capacity limits rather than data scarcity as the primary bottleneck. The findings inform practical resource allocation for stylistic generation in multimodal models and highlight the need to consider model capacity when aiming for stylistic saturation.

Abstract

Vision-language models are increasingly used to generate image captions in specific styles, such as humor or romantic. However, these transformer-based models often struggle with this subjective task in a zero-shot setting. While preference data can be used to align them toward a desired style, such data is expensive to acquire, limiting the ability to explore the models' full capabilities. This work addresses this by studying the data efficiency of aligning small vision-language models to humor and romantic styles. This approach helps to define the performance limits of these models and determine how little preference data is needed to achieve stylistic saturation, benchmarking their capabilities and limitations.

Figures (1)

• Figure 1: Data efficiency curves for stylistic alignment.