Does ChatGPT Have a Poetic Style?

TL;DR

The results show that GPT poetry is much more constrained and uniform than human poetry, showing a strong penchant for rhyme, quatrains, iambic meter, first-person plural perspectives (the authors, us, their), and specific vocabulary like"heart,""embrace,""echo,"and"whisper."

Abstract

Generating poetry has become a popular application of LLMs, perhaps especially of OpenAI's widely-used chatbot ChatGPT. What kind of poet is ChatGPT? Does ChatGPT have its own poetic style? Can it successfully produce poems in different styles? To answer these questions, we prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 different poetic forms and styles, about 40 different subjects, and in response to 3 different writing prompt templates. We then analyze the resulting 5.7k poems, comparing them to a sample of 3.7k poems from the Poetry Foundation and the Academy of American Poets. We find that the GPT models, especially GPT-4, can successfully produce poems in a range of both common and uncommon English-language forms in superficial yet noteworthy ways, such as by producing poems of appropriate lengths for sonnets (14 lines), villanelles (19 lines), and sestinas (39 lines). But the GPT models also exhibit their own distinct stylistic tendencies, both within and outside of these specific forms. Our results show that GPT poetry is much more constrained and uniform than human poetry, showing a strong penchant for rhyme, quatrains (4-line stanzas), iambic meter, first-person plural perspectives (we, us, our), and specific vocabulary like "heart," "embrace," "echo," and "whisper."

Figures (8)

• Figure 1: "Write a poem about the subject of social commentaries in the following form or style: limerick." An example poetry generation prompt and response by GPT-4, representing common tendencies of the model. While GPT-4 presents a comedic, topical take on a social commentary, it does not produce a typical limerick (usually 5 lines, anapestic meter, AABBA rhyme) but instead produces five quatrains with mostly iambic meter and AABB rhyme, what we suggest is its "default" mode.
• Figure 2: These boxplots represent the distribution of line lengths for poems with conventionally fixed lengths produced by GPT-3.5, GPT-4, and authors from the Poetry Foundation and the Academy of American Poets. The GPT models were also prompted with the generic style of "a poem"; to provide a comparison for the human poems, we include an aggregation of all poems from the sample. The boxes show the "interquartile range" (25% quartile-75% quartile) with a thicker line indicating the median average; the whiskers extend beyond the boxes by 1.5 times the IQR; the outliers are values that fall beyond the whiskers. The dotted red line indicates the expected number of lines for each form, e.g., a sonnet typically has 14 lines.
• Figure 3: These heatmaps represent the distribution of words, lines, and line breaks for fixed form poems by GPT-3.5, GPT-4, and authors from the Poetry Foundation and the Academy of American Poets. Darker squares represent a higher concentration of words and lines in specific positions across the poems; lighter squares represent a higher concentration of white space and line breaks. The GPT models are also prompted with the generic style of "a poem"; to provide a comparison for the human poems, we include an aggregation of all poems from the sample.
• Figure 4: These heatmaps represent the distribution of words, lines, and line breaks for all poems by GPT-3.5, GPT-4, and authors from the Poetry Foundation and the Academy of American Poets. Darker squares represent a higher concentration of words and lines in specific positions across the poems; lighter squares represent a higher concentration of white space and line breaks. The unusual dominance of quatrains (line breaks after 4 consecutive lines) is evident in the GPT-generated poems.
• Figure 5: The normalized frequency of pronouns used in poems by GPT-3.5, GPT-4, and authors from the Poetry Foundation and the Academy of American Poets, expressed per 100 words. The dotted line indicates normalized frequency in the GPT poems with the "holiday" and "occasion" poems removed (showing that first-person plural in the GPT-generated poems decreases slightly, and third-person increases slightly).