Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians

Piotr Wojciech Mirowski, Juliette Love, Kory W. Mathewson, Shakir Mohamed

TL;DR

This study evaluates whether instruction-tuned large language models can effectively support comedy writing by professional comedians. Through a mixed-methods design—live writing sessions, CSI-based evaluation, and focus-group discussions with 20 comedians—the authors reveal that while LLMs can quickly generate structure and content scaffolds, their humor quality is often bland, and safety filters can suppress edgy material and minority perspectives. The results highlight profound concerns about global cultural value alignment, data ownership, and the necessity of context-rich, community-driven approaches to tailor AI tools for artists. The paper argues for artist-led data governance, openness to open-source, and the integration of relational context to enable ethical, useful, and representative creativity-support tools for the performing arts. These findings inform responsible design and governance of AI in creative practice, emphasizing practical mitigations and the importance of artist communities in shaping tool behavior and norms.

Abstract

We interviewed twenty professional comedians who perform live shows in front of audiences and who use artificial intelligence in their artistic process as part of 3-hour workshops on ``AI x Comedy'' conducted at the Edinburgh Festival Fringe in August 2023 and online. The workshop consisted of a comedy writing session with large language models (LLMs), a human-computer interaction questionnaire to assess the Creativity Support Index of AI as a writing tool, and a focus group interrogating the comedians' motivations for and processes of using AI, as well as their ethical concerns about bias, censorship and copyright. Participants noted that existing moderation strategies used in safety filtering and instruction-tuned LLMs reinforced hegemonic viewpoints by erasing minority groups and their perspectives, and qualified this as a form of censorship. At the same time, most participants felt the LLMs did not succeed as a creativity support tool, by producing bland and biased comedy tropes, akin to ``cruise ship comedy material from the 1950s, but a bit less racist''. Our work extends scholarship about the subtle difference between, one the one hand, harmful speech, and on the other hand, ``offensive'' language as a practice of resistance, satire and ``punching up''. We also interrogate the global value alignment behind such language models, and discuss the importance of community-based value alignment and data ownership to build AI tools that better suit artists' needs.

A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians

TL;DR

This study evaluates whether instruction-tuned large language models can effectively support comedy writing by professional comedians. Through a mixed-methods design—live writing sessions, CSI-based evaluation, and focus-group discussions with 20 comedians—the authors reveal that while LLMs can quickly generate structure and content scaffolds, their humor quality is often bland, and safety filters can suppress edgy material and minority perspectives. The results highlight profound concerns about global cultural value alignment, data ownership, and the necessity of context-rich, community-driven approaches to tailor AI tools for artists. The paper argues for artist-led data governance, openness to open-source, and the integration of relational context to enable ethical, useful, and representative creativity-support tools for the performing arts. These findings inform responsible design and governance of AI in creative practice, emphasizing practical mitigations and the importance of artist communities in shaping tool behavior and norms.

Abstract

We interviewed twenty professional comedians who perform live shows in front of audiences and who use artificial intelligence in their artistic process as part of 3-hour workshops on ``AI x Comedy'' conducted at the Edinburgh Festival Fringe in August 2023 and online. The workshop consisted of a comedy writing session with large language models (LLMs), a human-computer interaction questionnaire to assess the Creativity Support Index of AI as a writing tool, and a focus group interrogating the comedians' motivations for and processes of using AI, as well as their ethical concerns about bias, censorship and copyright. Participants noted that existing moderation strategies used in safety filtering and instruction-tuned LLMs reinforced hegemonic viewpoints by erasing minority groups and their perspectives, and qualified this as a form of censorship. At the same time, most participants felt the LLMs did not succeed as a creativity support tool, by producing bland and biased comedy tropes, akin to ``cruise ship comedy material from the 1950s, but a bit less racist''. Our work extends scholarship about the subtle difference between, one the one hand, harmful speech, and on the other hand, ``offensive'' language as a practice of resistance, satire and ``punching up''. We also interrogate the global value alignment behind such language models, and discuss the importance of community-based value alignment and data ownership to build AI tools that better suit artists' needs.
Paper Structure (66 sections, 1 figure)

This paper contains 66 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Motivation: investigate the potential and implications of LLMs for comedy writing
  3. Using LLMs for humour, a task with human-level difficulty
  4. The utility of instruction-tuned LLMs as Creativity Support Tools
  5. Socio-Technical Systems concerns with LLMs for creative writing
  6. Investigating hypotheses about using LLMs to write comedy
  7. Biases in large language models
  8. Potential censorship of speech labeled as "offensive"
  9. Missing context
  10. Homogeneisation
  11. Investigating hypotheses via a mixed-methods study
  12. Methods
  13. Writing exercise
  14. Creativity Support Tools evaluation
  15. Focus Group Questions
  16. ...and 51 more sections

Figures (1)

  • Figure 1: Left: Evaluation of the instruction-tuned LLMs as creativity support tool for writing comedy using a Likert scale; each row corresponds to a question in a survey (questions are listed in Appendix \ref{['app:experience']}). Only 10 participants responded to question "Collaborated with AI". Right box plots (box for quartiles and whiskers for min and max) show the break-down of the Creativity Support Index cherry2014quantifying, respectively by a) previous usage of AI for writing comedy material, b) language used during the writing session with LLMs, c) the degree of familiarity with AI and d) the type of comedian or performer (stand-up vs. improv, theatre, film, dance etc.).