Dialogue with the Machine and Dialogue with the Art World: Evaluating Generative AI for Culturally-Situated Creativity

TL;DR

The paper addresses the problem of evaluating generative AI for culturally-situated creativity, arguing traditional benchmarks miss social and political contexts. It proposes a dialogic evaluation method combining 'dialogue with art worlds' and 'dialogue with machines', demonstrated in a Persian Gulf case study with three artists and commentators. The approach surfaces pathways like decentralized datasets and radical representational use-cases, and it reveals design implications to support diverse art worlds. The findings advocate for inclusive, socio-political AI design that accounts for community expertise and cultural contexts.

Abstract

This paper proposes dialogue as a method for evaluating generative AI tools for culturally-situated creative practice, that recognizes the socially situated nature of art. Drawing on sociologist Howard Becker's concept of Art Worlds, this method expands the scope of traditional AI and creativity evaluations beyond benchmarks, user studies with crowd-workers, or focus groups conducted with artists. Our method involves two mutually informed dialogues: 1) 'dialogues with art worlds' placing artists in conversation with experts such as art historians, curators, and archivists, and 2)'dialogues with the machine,' facilitated through structured artist- and critic-led experimentation with state-of-the-art generative AI tools. We demonstrate the value of this method through a case study with artists and experts steeped in non-western art worlds, specifically the Persian Gulf. We trace how these dialogues help create culturally rich and situated forms of evaluation for representational possibilities of generative AI that mimic the reception of generative artwork in the broader art ecosystem. Putting artists in conversation with commentators also allow artists to shift their use of the tools to respond to their cultural and creative context. Our study can provide generative AI researchers an understanding of the complex dynamics of technology, human creativity and the socio-politics of art worlds, to build more inclusive machines for diverse art worlds.

Figures (1)

• Figure 1: Overview of the dialogue-based approach followed in this study.