Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry

Bolei Ma, Yina Yao, Anna-Carolina Haensch

TL;DR

The paper investigates large language models' capabilities and evaluation biases in generating and assessing Tang poetry, a highly structured form of classical Chinese verse. It introduces a hybrid three-step framework combining automated feature analysis, LLM-based cross-evaluation, and human expert validation, applied across six open-source LLMs and five poetic dimensions. The study reveals systematic echo-chamber biases in LLM judgments, strong semantic alignment but shallow cultural depth, and notable misalignment with human expert judgments, highlighting limitations of automated metrics in culturally nuanced creative tasks. The proposed framework offers a transferable methodology for robust evaluation of AI in constrained literary domains and underscores the continued need for hybrid human–AI validation and diverse metrics to capture formal fidelity, imagery, and cultural authenticity.

Abstract

Large Language Models (LLMs) are increasingly applied to creative domains, yet their performance in classical Chinese poetry generation and evaluation remains poorly understood. We propose a three-step evaluation framework that combines computational metrics, LLM-as-a-judge assessment, and human expert validation. Using this framework, we evaluate six state-of-the-art LLMs across multiple dimensions of poetic quality, including themes, emotions, imagery, form, and style. Our analysis reveals systematic generation and evaluation biases: LLMs exhibit "echo chamber" effects when assessing creative quality, often converging on flawed standards that diverge from human judgments. These findings highlight both the potential and limitations of current capabilities of LLMs as proxy for literacy generation and the limited evaluation practices, thereby demonstrating the continued need of hybrid validation from both humans and models in culturally and technically complex creative tasks.

Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry

TL;DR

The paper investigates large language models' capabilities and evaluation biases in generating and assessing Tang poetry, a highly structured form of classical Chinese verse. It introduces a hybrid three-step framework combining automated feature analysis, LLM-based cross-evaluation, and human expert validation, applied across six open-source LLMs and five poetic dimensions. The study reveals systematic echo-chamber biases in LLM judgments, strong semantic alignment but shallow cultural depth, and notable misalignment with human expert judgments, highlighting limitations of automated metrics in culturally nuanced creative tasks. The proposed framework offers a transferable methodology for robust evaluation of AI in constrained literary domains and underscores the continued need for hybrid human–AI validation and diverse metrics to capture formal fidelity, imagery, and cultural authenticity.

Abstract

Large Language Models (LLMs) are increasingly applied to creative domains, yet their performance in classical Chinese poetry generation and evaluation remains poorly understood. We propose a three-step evaluation framework that combines computational metrics, LLM-as-a-judge assessment, and human expert validation. Using this framework, we evaluate six state-of-the-art LLMs across multiple dimensions of poetic quality, including themes, emotions, imagery, form, and style. Our analysis reveals systematic generation and evaluation biases: LLMs exhibit "echo chamber" effects when assessing creative quality, often converging on flawed standards that diverge from human judgments. These findings highlight both the potential and limitations of current capabilities of LLMs as proxy for literacy generation and the limited evaluation practices, thereby demonstrating the continued need of hybrid validation from both humans and models in culturally and technically complex creative tasks.
Paper Structure (37 sections, 5 equations, 34 figures, 8 tables)

This paper contains 37 sections, 5 equations, 34 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Experimental Design and Methods
  4. Generation
  5. Evaluation
  6. Evaluation Pipeline
  7. Evaluation Methods (three-step analysis)
  8. Step 1: Computational feature extraction.
  9. Step 2: Descriptive analysis of LLM-as-judge scores.
  10. Step 3: Human-AI alignment.
  11. Experiments and Results
  12. Computational Feature Analysis
  13. Lexical Fingerprinting
  14. Lexical Diversity and Predictability
  15. Semantic and Cultural Coherence
  16. ...and 22 more sections

Figures (34)

  • Figure 1: The basic framework of poetry generation and evaluation.
  • Figure 2: Top 10 keywords for all models, with highest TF-IDF scores marked red for each model.
  • Figure 3: Entropy scores across models.
  • Figure 4: Entropy scores across models by theme.
  • Figure 5: Semantic distinction by theme.
  • ...and 29 more figures