Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Diversity in Automatic Poetry Generation

Yanran Chen, Hannes Gröner, Sina Zarrieß, Steffen Eger

TL;DR

It is found that current automatic poetry systems are considerably underdiverse along multiple dimensions — they often do not rhyme sufficiently, are semantically too uniform and even do not match the length distribution of human poetry.

Abstract

Natural Language Generation (NLG), and more generally generative AI, are among the currently most impactful research fields. Creative NLG, such as automatic poetry generation, is a fascinating niche in this area. While most previous research has focused on forms of the Turing test when evaluating automatic poetry generation -- can humans distinguish between automatic and human generated poetry -- we evaluate the diversity of automatically generated poetry (with a focus on quatrains), by comparing distributions of generated poetry to distributions of human poetry along structural, lexical, semantic and stylistic dimensions, assessing different model types (word vs. character-level, general purpose LLMs vs. poetry-specific models), including the very recent LLaMA3-8B, and types of fine-tuning (conditioned vs. unconditioned). We find that current automatic poetry systems are considerably underdiverse along multiple dimensions -- they often do not rhyme sufficiently, are semantically too uniform and even do not match the length distribution of human poetry. Our experiments reveal, however, that style-conditioning and character-level modeling clearly increases diversity across virtually all dimensions we explore. Our identified limitations may serve as the basis for more genuinely diverse future poetry generation models.

Evaluating Diversity in Automatic Poetry Generation

TL;DR

It is found that current automatic poetry systems are considerably underdiverse along multiple dimensions — they often do not rhyme sufficiently, are semantically too uniform and even do not match the length distribution of human poetry.

Abstract

Natural Language Generation (NLG), and more generally generative AI, are among the currently most impactful research fields. Creative NLG, such as automatic poetry generation, is a fascinating niche in this area. While most previous research has focused on forms of the Turing test when evaluating automatic poetry generation -- can humans distinguish between automatic and human generated poetry -- we evaluate the diversity of automatically generated poetry (with a focus on quatrains), by comparing distributions of generated poetry to distributions of human poetry along structural, lexical, semantic and stylistic dimensions, assessing different model types (word vs. character-level, general purpose LLMs vs. poetry-specific models), including the very recent LLaMA3-8B, and types of fine-tuning (conditioned vs. unconditioned). We find that current automatic poetry systems are considerably underdiverse along multiple dimensions -- they often do not rhyme sufficiently, are semantically too uniform and even do not match the length distribution of human poetry. Our experiments reveal, however, that style-conditioning and character-level modeling clearly increases diversity across virtually all dimensions we explore. Our identified limitations may serve as the basis for more genuinely diverse future poetry generation models.
Paper Structure (23 sections, 2 figures, 6 tables)

This paper contains 23 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diversity
  4. Poetry generation
  5. Diversity in Poetry Generation
  6. Memorization.
  7. Poem length.
  8. Rhyme patterns.
  9. Lexical diversity.
  10. Semantic diversity.
  11. Experiment Setup
  12. Data
  13. Models
  14. Evaluation
  15. Diversity Evaluation
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Distribution of rhyme schemes in (a) the human data, and the samples from the (b) best, (c) worst, and (d) average models based on their KL divergence from the human distribution for German.
  • Figure 2: Distribution of rhyme schemes in (a) the human data, and the samples from the (b) best, (c) worst, and (d) average models based on their KL divergence from the human distribution for English.