Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-modal Learnable Queries for Image Aesthetics Assessment

Zhiwei Xiong, Yunfan Zhang, Zhiqi Shen, Peiran Ren, Han Yu

TL;DR

This work tackles the intrinsic subjectivity of image aesthetics by leveraging user comments alongside images. It introduces Multi-modal Learnable Queries (MMLQ), which uses learnable queries to extract cross-modal aesthetic features from frozen vision and language encoders through a stack of Multi-modal Interaction Blocks (MMIBs). The approach, trained with Earth Mover’s Distance loss, achieves state-of-the-art performance on the AVA dataset with AVA-Comments, significantly outperforming prior methods. The results highlight the value of integrating multi-modal pre-trained representations for IAA and point to future directions in captioning and retrieval to further reduce reliance on explicit comments.

Abstract

Image aesthetics assessment (IAA) is attracting wide interest with the prevalence of social media. The problem is challenging due to its subjective and ambiguous nature. Instead of directly extracting aesthetic features solely from the image, user comments associated with an image could potentially provide complementary knowledge that is useful for IAA. With existing large-scale pre-trained models demonstrating strong capabilities in extracting high-quality transferable visual and textual features, learnable queries are shown to be effective in extracting useful features from the pre-trained visual features. Therefore, in this paper, we propose MMLQ, which utilizes multi-modal learnable queries to extract aesthetics-related features from multi-modal pre-trained features. Extensive experimental results demonstrate that MMLQ achieves new state-of-the-art performance on multi-modal IAA, beating previous methods by 7.7% and 8.3% in terms of SRCC and PLCC, respectively.

Multi-modal Learnable Queries for Image Aesthetics Assessment

TL;DR

This work tackles the intrinsic subjectivity of image aesthetics by leveraging user comments alongside images. It introduces Multi-modal Learnable Queries (MMLQ), which uses learnable queries to extract cross-modal aesthetic features from frozen vision and language encoders through a stack of Multi-modal Interaction Blocks (MMIBs). The approach, trained with Earth Mover’s Distance loss, achieves state-of-the-art performance on the AVA dataset with AVA-Comments, significantly outperforming prior methods. The results highlight the value of integrating multi-modal pre-trained representations for IAA and point to future directions in captioning and retrieval to further reduce reliance on explicit comments.

Abstract

Image aesthetics assessment (IAA) is attracting wide interest with the prevalence of social media. The problem is challenging due to its subjective and ambiguous nature. Instead of directly extracting aesthetic features solely from the image, user comments associated with an image could potentially provide complementary knowledge that is useful for IAA. With existing large-scale pre-trained models demonstrating strong capabilities in extracting high-quality transferable visual and textual features, learnable queries are shown to be effective in extracting useful features from the pre-trained visual features. Therefore, in this paper, we propose MMLQ, which utilizes multi-modal learnable queries to extract aesthetics-related features from multi-modal pre-trained features. Extensive experimental results demonstrate that MMLQ achieves new state-of-the-art performance on multi-modal IAA, beating previous methods by 7.7% and 8.3% in terms of SRCC and PLCC, respectively.
Paper Structure (14 sections, 8 equations, 6 figures, 4 tables)

This paper contains 14 sections, 8 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. The Proposed Approach
  4. Multi-modal Pre-trained Features
  5. Multi-modal Interaction Block
  6. Prediction Head for IAA
  7. Experimental Evaluation
  8. Dataset
  9. Experiment Settings
  10. Comparison with State-of-the-art Methods
  11. Ablation Studies
  12. Example Results
  13. Limitation
  14. Conlusion and Future Work

Figures (6)

  • Figure 1: Example images with their corresponding image ID, ground truth MOS, and selected user comments from the AVA murray2012ava and AVA-Comments zhou2016joint datasets.
  • Figure 2: The overall structure of the proposed multi-modal learnable queries (MMLQ) method for IAA. The visual and textual LQs learn visual and textual aesthetic features through a replaceable self-attention layer, a separate multi-modal cross-attention layer with corresponding visual and textual pre-trained features, and a replaceable feed-forward layer. Finally, the averaged visual LQs and the averaged textual LQs are concatenated and fed into linear layers followed by Softmax to output the estimated aesthetic DOS of the input image.
  • Figure 3: The histogram of the accumulated comment lengths for each image in AVA-Comments zhou2016joint. The vertical dotted line in red indicates a comment length of 512, which is the maximum comment length ($N_w$) allowed in our experiments.
  • Figure 4: Performance with different numbers of MMIBs.
  • Figure 5: Performance with different numbers of LQs.
  • ...and 1 more figures