Aesthetics as Structural Harm: Algorithmic Lookism Across Text-to-Image Generation and Classification

TL;DR

This paper demonstrates that algorithmic lookism—systematically linking facial attractiveness to positive traits—operates as an infrastructural bias across text-to-image generation and downstream gender classification. Using 26,400 synthetic faces from Stable Diffusion 2.1 and 3.5 Medium, it shows consistent associations between attractiveness and attributes like happiness, intelligence, sociability, and trustworthiness, with amplified effects for women and certain racial groups. It further reveals gender classification disparities that worsen for female faces, especially under negative attributes, and identifies qualitative patterns such as age homogenization, happiness-beauty conflation, gendered exposure, and geographic reductionism that deepen representation and recognition harms. The study links these patterns to neoliberal rationality and orientalist regimes, arguing that newer models intensify aesthetic constraints despite improved data curation, thereby privatizing harm and reinforcing social inequalities in AI systems.

Abstract

This paper examines algorithmic lookism-the systematic preferential treatment based on physical appearance-in text-to-image (T2I) generative AI and a downstream gender classification task. Through the analysis of 26,400 synthetic faces created with Stable Diffusion 2.1 and 3.5 Medium, we demonstrate how generative AI models systematically associate facial attractiveness with positive attributes and vice-versa, mirroring socially constructed biases rather than evidence-based correlations. Furthermore, we find significant gender bias in three gender classification algorithms depending on the attributes of the input faces. Our findings reveal three critical harms: (1) the systematic encoding of attractiveness-positive attribute associations in T2I models; (2) gender disparities in classification systems, where women's faces, particularly those generated with negative attributes, suffer substantially higher misclassification rates than men's; and (3) intensifying aesthetic constraints in newer models through age homogenization, gendered exposure patterns, and geographic reductionism. These convergent patterns reveal algorithmic lookism as systematic infrastructure operating across AI vision systems, compounding existing inequalities through both representation and recognition. Disclaimer: This work includes visual and textual content that reflects stereotypical associations between physical appearance and socially constructed attributes, including gender, race, and traits associated with social desirability. Any such associations found in this study emerge from the biases embedded in generative AI systems-not from empirical truths or the authors' views.

Figures (15)

• Figure 1: The attractiveness halo effect (top) guides dataset generation to test algorithmic lookism across (a) generation, (b) classification, and (c) aesthetic model refinement in SD 3.5 vs. 2.1. Images in the example are for images generated of White women and men.
• Figure 2: Distributional measure scores ($L_g^{(a)}$) across demographic groups in images generated with Stable Diffusion 2.1 (left) and 3.5 (right), using CLIP embeddings. Negative values (blue) indicate closeness to unattractive faces; positive values (red) to attractive ones. An asterisk (*) indicates statistical significance ($p<0.05$). The neutral trait is highlighted with a pink box. The results for men/women are highlighted with a green/yellow box, respectively. A = Attractiveness, H = Happiness, I = Intelligence, S = Sociability, T = Trustworthiness, N = Neutral.
• Figure 3: Distributional measure scores ($L_g^{(a)}$) scores across demographic groups in images generated with Stable Diffusion 2.1 (left) and 3.5 (right), using ArcFace embeddings. Negative values (blue) indicate closeness to unattractive faces; positive values (red) to attractive ones. An asterisk (*) indicates statistical significance ($p<0.05$). The neutral trait is highlighted with a pink box. The results for men/women are highlighted with a green/yellow box, respectively. A = Attractiveness, H = Happiness, I = Intelligence, S = Sociability, T = Trustworthiness, N = Neutral.
• Figure 4: Heatmaps of gender classification accuracy (Mean ± Std) for InsightFace, DeepFace, and FairFace for SD 2.1 (orange) and SD 3.5 (purple) . A = Attractiveness, H = Happiness, I = Intelligence, S = Sociability, T = Trustworthiness. Performance corresponding to female (Women)/male (Men) faces is highlighed in Yellow • / Green •, respectively. Accuracies in classifying neutral faces SD 2.1 (%): InsightFace: (W) 80.9 ± 10.7 (M) 87.5 ± 1.5, DeepFace: (W) 51.1 ± 9.0 (M) 99.6 ± 0.1, FairFace: (W) 96.8 ± 1.4 (M) 98.9±0.4. Accuracies in classifying neutral faces SD 3.5 (%): InsightFace: (W) 83.0 ± 17.5 (M) 89.8 ± 5.7, DeepFace: (W) 76.5 ± 14.3 (W) 100.0 ± 0, FairFace: (W) 99.8 ± 0.2 (M) 100 ± 0.
• Figure 5: (a) Age homogeneization and agism in the generated images: Age distribution of synthetic faces by gender and associated attributes in SD 2.1 and SD 3.5. The images created with positive attributes tend to depict younger individuals than the images created with negative or neutral attributes. Age classification was performed using FairFace. Top/bottom rows correspond to images of women/men, respectively. (b) Generated faces from SD 2.1 (orange) and SD 3.5 (purple) for the "Neutral" prompt for "White" and "Black" categories, illustrating the shift toward visually younger appearances.