Scaling Up Personalized Image Aesthetic Assessment via Task Vector Customization

TL;DR

This work presents a unique approach that leverages readily available databases for general image aesthetic assessment and image quality assessment and views each database as a distinct image score regression task that exhibits varying degrees of personalization potential.

Abstract

The task of personalized image aesthetic assessment seeks to tailor aesthetic score prediction models to match individual preferences with just a few user-provided inputs. However, the scalability and generalization capabilities of current approaches are considerably restricted by their reliance on an expensive curated database. To overcome this long-standing scalability challenge, we present a unique approach that leverages readily available databases for general image aesthetic assessment and image quality assessment. Specifically, we view each database as a distinct image score regression task that exhibits varying degrees of personalization potential. By determining optimal combinations of task vectors, known to represent specific traits of each database, we successfully create personalized models for individuals. This approach of integrating multiple models allows us to harness a substantial amount of data. Our extensive experiments demonstrate the effectiveness of our approach in generalizing to previously unseen domains-a challenge previous approaches have struggled to achieve-making it highly applicable to real-world scenarios. Our novel approach significantly advances the field by offering scalable solutions for personalized aesthetic assessment and establishing high standards for future research. https://yeolj00.github.io/personal-projects/personalized-aesthetics/

Figures (13)

• Figure 1: Comparison between PIAA and other image regression databases. Our approach has the flexibility to leverage any image regression databases.
• Figure 2: Zero-shot personalization performance for four users in the Flickr-AES database flickeraes. Each colored bar refers to the personalization performance of a model trained on a specific database.
• Figure 3: (a) Multiple models are fine-tuned from a single pre-trained model on different tasks to obtain task vectors. For simplicity, we omit the discussion of layers. (b) Given a small number of user-provided samples, the coefficients corresponding to each task vector are optimized with the rank loss.
• Figure 4: Similarity matrix between task vectors derived from each database. The similarities are measured with the cosine similarity.
• Figure 5: Performance gains with respect to the number of databases used. The dotted lines indicate the previous state-of-the-art performance.