Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception

Yipo Huang, Xiangfei Sheng, Zhichao Yang, Quan Yuan, Zhichao Duan, Pengfei Chen, Leida Li, Weisi Lin, Guangming Shi

TL;DR

This work addresses the challenge of enabling multimodal foundation models to perceive image aesthetics by building AesMMIT, a corpus-rich, human-annotated instruction-tuning dataset derived from 88K aesthetic critiques on 21,904 images and expanded to 409K multi-typed instructions via GPT-4 refinement. Using AesMMIT, the authors fine-tune open-source MLLMs to create AesExpert, which achieves state-of-the-art aesthetics perception surpassing GPT-4V and Gemini-Pro-Vision on comprehensive AesBench metrics. The approach significantly improves across dimensions like aesthetic perception, empathy, assessment, and interpretation, with notable gains on composition and AI-generated imagery. The authors release the dataset, AesExpert models, and a visual chatbot demo to stimulate further research and practical applications in smart photography, album management, and image enhancement.

Abstract

The highly abstract nature of image aesthetics perception (IAP) poses significant challenge for current multimodal large language models (MLLMs). The lack of human-annotated multi-modality aesthetic data further exacerbates this dilemma, resulting in MLLMs falling short of aesthetics perception capabilities. To address the above challenge, we first introduce a comprehensively annotated Aesthetic Multi-Modality Instruction Tuning (AesMMIT) dataset, which serves as the footstone for building multi-modality aesthetics foundation models. Specifically, to align MLLMs with human aesthetics perception, we construct a corpus-rich aesthetic critique database with 21,904 diverse-sourced images and 88K human natural language feedbacks, which are collected via progressive questions, ranging from coarse-grained aesthetic grades to fine-grained aesthetic descriptions. To ensure that MLLMs can handle diverse queries, we further prompt GPT to refine the aesthetic critiques and assemble the large-scale aesthetic instruction tuning dataset, i.e. AesMMIT, which consists of 409K multi-typed instructions to activate stronger aesthetic capabilities. Based on the AesMMIT database, we fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, dubbed AesExpert. Extensive experiments demonstrate that the proposed AesExpert models deliver significantly better aesthetic perception performances than the state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro-Vision. Project homepage: https://yipoh.github.io/aes-expert/.

AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception

TL;DR

This work addresses the challenge of enabling multimodal foundation models to perceive image aesthetics by building AesMMIT, a corpus-rich, human-annotated instruction-tuning dataset derived from 88K aesthetic critiques on 21,904 images and expanded to 409K multi-typed instructions via GPT-4 refinement. Using AesMMIT, the authors fine-tune open-source MLLMs to create AesExpert, which achieves state-of-the-art aesthetics perception surpassing GPT-4V and Gemini-Pro-Vision on comprehensive AesBench metrics. The approach significantly improves across dimensions like aesthetic perception, empathy, assessment, and interpretation, with notable gains on composition and AI-generated imagery. The authors release the dataset, AesExpert models, and a visual chatbot demo to stimulate further research and practical applications in smart photography, album management, and image enhancement.

Abstract

The highly abstract nature of image aesthetics perception (IAP) poses significant challenge for current multimodal large language models (MLLMs). The lack of human-annotated multi-modality aesthetic data further exacerbates this dilemma, resulting in MLLMs falling short of aesthetics perception capabilities. To address the above challenge, we first introduce a comprehensively annotated Aesthetic Multi-Modality Instruction Tuning (AesMMIT) dataset, which serves as the footstone for building multi-modality aesthetics foundation models. Specifically, to align MLLMs with human aesthetics perception, we construct a corpus-rich aesthetic critique database with 21,904 diverse-sourced images and 88K human natural language feedbacks, which are collected via progressive questions, ranging from coarse-grained aesthetic grades to fine-grained aesthetic descriptions. To ensure that MLLMs can handle diverse queries, we further prompt GPT to refine the aesthetic critiques and assemble the large-scale aesthetic instruction tuning dataset, i.e. AesMMIT, which consists of 409K multi-typed instructions to activate stronger aesthetic capabilities. Based on the AesMMIT database, we fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, dubbed AesExpert. Extensive experiments demonstrate that the proposed AesExpert models deliver significantly better aesthetic perception performances than the state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro-Vision. Project homepage: https://yipoh.github.io/aes-expert/.
Paper Structure (17 sections, 3 figures, 7 tables)

This paper contains 17 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-modality Foundation Model
  4. Multimodal Instruction-following Dataset
  5. Dataset Construction
  6. Experiment Preparation
  7. Collecting Human Aesthetic Feedback
  8. Generating Instruction-following Data
  9. Model Fine-tuning
  10. Model Architecture
  11. Supervised Fine-Tuning
  12. Experiments
  13. Implementation Details
  14. Performance Comparison
  15. Performance Improvement
  16. ...and 2 more sections

Figures (3)

  • Figure 1: The proposed dataset construction pipeline. First, we filter 21K diverse-sourced images based on scene, quality and source, and invite 48 human subjects who met the comprehensive criteria. Then, we collect 88K human feedbacks on image aesthetics perception. Finally, human feedbacks are converted into 409K instruction-following pairs (the AesMMIT dataset).
  • Figure 2: Statistics of top-20 frequently-used words in the AesMMIT database.
  • Figure 3: Performance comparison using different datasets.