CCiV: A Benchmark for Structure, Rhythm and Quality in LLM-Generated Chinese \textit{Ci} Poetry

TL;DR

This work introduces CCiV, a benchmark designed to assess LLM-generated classical Chinese poetry across these three dimensions: structure, rhythm, and quality, and shows that form-aware prompting can improve structural and tonal control for stronger models, while potentially degrading weaker ones.

Abstract

The generation of classical Chinese \textit{Ci} poetry, a form demanding a sophisticated blend of structural rigidity, rhythmic harmony, and artistic quality, poses a significant challenge for large language models (LLMs). To systematically evaluate and advance this capability, we introduce \textbf{C}hinese \textbf{Ci}pai \textbf{V}ariants (\textbf{CCiV}), a benchmark designed to assess LLM-generated \textit{Ci} poetry across these three dimensions: structure, rhythm, and quality. Our evaluation of 17 LLMs on 30 \textit{Cipai} reveals two critical phenomena: models frequently generate valid but unexpected historical variants of a poetic form, and adherence to tonal patterns is substantially harder than structural rules. We further show that form-aware prompting can improve structural and tonal control for stronger models, while potentially degrading weaker ones. Finally, we observe weak and inconsistent alignment between formal correctness and literary quality in our sample. CCiV highlights the need for variant-aware evaluation and more holistic constrained creative generation methods.

Figures (11)

• Figure 1: Prompting strategies for Ci poetry generation: direct prompting versus form-aware prompting with explicit structural guidance.
• Figure 2: Prompt template for evaluating Informativeness and Aesthetic metrics.
• Figure 3: Heatmap of structural accuracy against standard forms under direct prompting. Each row represents a model (ordered by overall performance), and each column represents a Cipai form (ordered by character count, right-to-left: 116 to 27). Darker cells indicate higher accuracy. Two patterns emerge: (1) better performance on shorter forms (left) than longer forms (right), and (2) stark performance gaps between model families. Anomalous low-accuracy columns (e.g., C27, C28) reflect the variant generation phenomenon.
• Figure 4: Direct prompt example (zero-shot). No additional structural guidance is provided. Text in green is example variable and text in red is example output.
• Figure 5: Form-aware prompt example. Explicit structural guidance is provided. Text in green is example variable and text in red is example output.