Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Algorithmic Gaze: An Audit and Ethnography of the LAION-Aesthetics Predictor Model

Jordan Taylor, William Agnew, Maarten Sap, Sarah E. Fox, Haiyi Zhu

TL;DR

The paper investigates the LAION-Aesthetics Predictor (LAP), a widely used aesthetic evaluator for curating training data and assessing AI-generated images, and reveals that LAP encodes imperial, realist, and male visual biases. Through audits on the LAION-Aesthetics Dataset, the MET collection, and WikiArt, complemented by a digital ethnography of LAP's development, the authors show systematic biases: higher scores for Western/Japanese art and for captions mentioning women, with underrepresentation or exclusion of non-Western art and men or LGBTQ+ references. They trace these biases to the founder's tastes, uneven dataset documentation, and the mixing of diverse aesthetic measures, arguing that the current approach to aesthetic evaluation reinforces representational harms. The work advocates for pluralistic, descriptive evaluation approaches and a socio-technical audit+ethnography methodology to surface culture-specific values and guide more inclusive AI evaluation practices.

Abstract

Visual generative AI models are trained using a one-size-fits-all measure of aesthetic appeal. However, what is deemed "aesthetic" is inextricably linked to personal taste and cultural values, raising the question of whose taste is represented in visual generative AI models. In this work, we study an aesthetic evaluation model--LAION Aesthetic Predictor (LAP)--that is widely used to curate datasets to train visual generative image models, like Stable Diffusion, and evaluate the quality of AI-generated images. To understand what LAP measures, we audited the model across three datasets. First, we examined the impact of aesthetic filtering on the LAION-Aesthetics Dataset (approximately 1.2B images), which was curated from LAION-5B using LAP. We find that the LAP disproportionally filters in images with captions mentioning women, while filtering out images with captions mentioning men or LGBTQ+ people. Then, we used LAP to score approximately 330k images across two art datasets, finding the model rates realistic images of landscapes, cityscapes, and portraits from western and Japanese artists most highly. In doing so, the algorithmic gaze of this aesthetic evaluation model reinforces the imperial and male gazes found within western art history. In order to understand where these biases may have originated, we performed a digital ethnography of public materials related to the creation of LAP. We find that the development of LAP reflects the biases we found in our audits, such as the aesthetic scores used to train LAP primarily coming from English-speaking photographers and western AI-enthusiasts. In response, we discuss how aesthetic evaluation can perpetuate representational harms and call on AI developers to shift away from prescriptive measures of "aesthetics" toward more pluralistic evaluation.

The Algorithmic Gaze: An Audit and Ethnography of the LAION-Aesthetics Predictor Model

TL;DR

The paper investigates the LAION-Aesthetics Predictor (LAP), a widely used aesthetic evaluator for curating training data and assessing AI-generated images, and reveals that LAP encodes imperial, realist, and male visual biases. Through audits on the LAION-Aesthetics Dataset, the MET collection, and WikiArt, complemented by a digital ethnography of LAP's development, the authors show systematic biases: higher scores for Western/Japanese art and for captions mentioning women, with underrepresentation or exclusion of non-Western art and men or LGBTQ+ references. They trace these biases to the founder's tastes, uneven dataset documentation, and the mixing of diverse aesthetic measures, arguing that the current approach to aesthetic evaluation reinforces representational harms. The work advocates for pluralistic, descriptive evaluation approaches and a socio-technical audit+ethnography methodology to surface culture-specific values and guide more inclusive AI evaluation practices.

Abstract

Visual generative AI models are trained using a one-size-fits-all measure of aesthetic appeal. However, what is deemed "aesthetic" is inextricably linked to personal taste and cultural values, raising the question of whose taste is represented in visual generative AI models. In this work, we study an aesthetic evaluation model--LAION Aesthetic Predictor (LAP)--that is widely used to curate datasets to train visual generative image models, like Stable Diffusion, and evaluate the quality of AI-generated images. To understand what LAP measures, we audited the model across three datasets. First, we examined the impact of aesthetic filtering on the LAION-Aesthetics Dataset (approximately 1.2B images), which was curated from LAION-5B using LAP. We find that the LAP disproportionally filters in images with captions mentioning women, while filtering out images with captions mentioning men or LGBTQ+ people. Then, we used LAP to score approximately 330k images across two art datasets, finding the model rates realistic images of landscapes, cityscapes, and portraits from western and Japanese artists most highly. In doing so, the algorithmic gaze of this aesthetic evaluation model reinforces the imperial and male gazes found within western art history. In order to understand where these biases may have originated, we performed a digital ethnography of public materials related to the creation of LAP. We find that the development of LAP reflects the biases we found in our audits, such as the aesthetic scores used to train LAP primarily coming from English-speaking photographers and western AI-enthusiasts. In response, we discuss how aesthetic evaluation can perpetuate representational harms and call on AI developers to shift away from prescriptive measures of "aesthetics" toward more pluralistic evaluation.
Paper Structure (35 sections, 2 figures, 4 tables)

This paper contains 35 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Background on LAION-Aesthetics
  4. AI Evaluation & Alignment
  5. Auditing LAION-Aesthetics Predictor Model
  6. Audit Methodology
  7. Findings: The LAION-Aesthetics Dataset
  8. The LAION-Aesthetics Predictor highly rates images from websites used by independent visual artists and photographers
  9. The LAION-Aesthetics Predictor is more likely to rate images with descriptions mentioning woman $\geq$ 6.5, while those mentioning men or LGBTQ+ people are more likely to be filtered out.
  10. Findings: The Metropolitan Museum of Art Dataset
  11. The LAION-Aesthetics Predictor rates western art more highly than art from the Majority World
  12. The LAION-Aesthetics Predictor rates paintings and photographs by western or Japanese artists most highly.
  13. Findings: The WikiArt Dataset
  14. The LAION-Aesthetics Predictor more highly rates realistic images of cityscapes, landscapes, and portraits, while less realistic art was rated lower.
  15. Findings: Qualitative Visual Analysis
  16. ...and 20 more sections

Figures (2)

  • Figure 1: Top 25 domains of images in the LAION-Aesthetics Dataset rated 6.5+ by the LAION-Aesthetic Prediction model
  • Figure 2: Pointwise Mutual Information (PMI) between regex and images being included in the 6.5+ subset of the LAION-Aesthetics Dataset. Terms with higher PMI (e.g., wom[ae]n) have higher likelihood of being included.