Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms

TL;DR

This work tackles the misalignment between vision-language retrieval models and human aesthetics by proposing an end-to-end alignment framework. It combines LLM-driven query rephrasing to inject aesthetic reasoning, LMM-based re-ranking, and a preference-based reinforcement learning objective (DPO) to distill guidance from both LLMs and aesthetic models into the vision-language backbone. Two novel benchmarks are introduced: HPIR for human preference evaluation and GPT-4V-based win-rate judgments to assess system-level aesthetic alignment. Experimental results show that LLM-informed queries and RL fine-tuning significantly improve aesthetic quality of retrieved results with minimal impact on traditional retrieval metrics, suggesting a practical path toward end-to-end aesthetic-aligned retrieval. The approach offers a generalizable direction for aligning visual systems with subtler user preferences beyond aesthetics, potentially reducing reliance on cascaded filter pipelines in large-scale deployments.

Abstract

Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibility. In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system. Advanced retrieval systems usually adopt a cascade of aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics. Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.

Figures (26)

• Figure 1: Alignment examples. W/o alignment, the models may prefer samples violating user intents.
• Figure 2: The concept of aesthetic, which inspires our pipeline of alignment.
• Figure 3: Effect of LLM rephrasing. All images are retrieved from the same fixed engine. The advancement of LLM rephrasing has clearly enhanced the aesthetic quality of outputs, particularly in expressing abstract notions and stylistic elements.
• Figure 4: An example illustration for the construction of partially ordered pair dataset.
• Figure 5: Qualitative comparison of top-4 retrieval results between models with and without our proposed alignment fine-tuning.