From job titles to jawlines: Using context voids to study generative AI systems

TL;DR

This work introduces a speculative design framework to study generative AI by creating context voids between textual bios (CVs) and visual outputs (headshots) to elicit system biases. By applying a two-stage GPT-4→DALL-E pipeline to academic CVs, the authors reveal how missing-context prompts can yield biased, stereotype-laden portraits, even when gender markers are removed. The study contributes a qualitative, framework-driven methodology for probing sensemaking and biases in multimodal AI systems, highlighting risks of representation bias and value drift across model cascades. While exploratory and limited in scope, the approach offers a flexible auditing lens for uncovering latent biases in closed, opaque systems and broader multimodal pipelines.

Abstract

In this paper, we introduce a speculative design methodology for studying the behavior of generative AI systems, framing design as a mode of inquiry. We propose bridging seemingly unrelated domains to generate intentional context voids, using these tasks as probes to elicit AI model behavior. We demonstrate this through a case study: probing the ChatGPT system (GPT-4 and DALL-E) to generate headshots from professional Curricula Vitae (CVs). In contrast to traditional ways, our approach assesses system behavior under conditions of radical uncertainty -- when forced to invent entire swaths of missing context -- revealing subtle stereotypes and value-laden assumptions. We qualitatively analyze how the system interprets identity and competence markers from CVs, translating them into visual portraits despite the missing context (i.e. physical descriptors). We show that within this context void, the AI system generates biased representations, potentially relying on stereotypical associations or blatant hallucinations.

Figures (2)

• Figure 1: Generation of headshots from CVs. An LLM (GPT-4) generates a prompt from a real academic CV (Phase 1) which is then used by a text-to-image generation model (DALL-E 3) to generate an image, associated with the CV (Phase 2).
• Figure 2: This figure presents AI-generated headshots based on academic CVs, highlighting gender biases in image generation. Two images were created per CV, with purple outlines indicating portraits classified as female (by the AI) and red outlines indicating portraits classified as male (by the AI). The analysis reveals a bias towards generating male images, regardless of the gender of the CV holder. Female portraits were more commonly generated when CVs contained clear gender markers or were linked to stereotypically female-dominated fields, such as art history. Generated prompts, and the AI-assigned gender for the original headshots of the corresponding CV holders, along with their primary field of study, are shown below the generated portraits. It is important to note that we refrain from making any gender inferences from the portraits themselves, whether original or generated. Additionally, since the generated images depict fictional individuals, they have no inherent gender; thus, we rely on the AI's perception of gender for alignment with our research questions.