Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry

Cheng Zhao, Bin Wang, Zhen Wang

TL;DR

This paper attempts to explore the mysteries of literary texts from the perspective of LLMs, using ancient Chinese poetry as an example for experimentation and observes a series of enlightening phenomena that provide technical support for the future development of high-level literary creation based on LLMs.

Abstract

The birth and rapid development of large language models (LLMs) have caused quite a stir in the field of literature. Once considered unattainable, AI's role in literary creation is increasingly becoming a reality. In genres such as poetry, jokes, and short stories, numerous AI tools have emerged, offering refreshing new perspectives. However, it's difficult to further improve the quality of these works. This is primarily because understanding and appreciating a good literary work involves a considerable threshold, such as knowledge of literary theory, aesthetic sensibility, interdisciplinary knowledge. Therefore, authoritative data in this area is quite lacking. Additionally, evaluating literary works is often complex and hard to fully quantify, which directly hinders the further development of AI creation. To address this issue, this paper attempts to explore the mysteries of literary texts from the perspective of LLMs, using ancient Chinese poetry as an example for experimentation. First, we collected a variety of ancient poems from different sources and had experts annotate a small portion of them. Then, we designed a range of comprehension metrics based on LLMs to evaluate all these poems. Finally, we analyzed the correlations and differences between various poem collections to identify literary patterns. Through our experiments, we observed a series of enlightening phenomena that provide technical support for the future development of high-level literary creation based on LLMs.

Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry

TL;DR

This paper attempts to explore the mysteries of literary texts from the perspective of LLMs, using ancient Chinese poetry as an example for experimentation and observes a series of enlightening phenomena that provide technical support for the future development of high-level literary creation based on LLMs.

Abstract

The birth and rapid development of large language models (LLMs) have caused quite a stir in the field of literature. Once considered unattainable, AI's role in literary creation is increasingly becoming a reality. In genres such as poetry, jokes, and short stories, numerous AI tools have emerged, offering refreshing new perspectives. However, it's difficult to further improve the quality of these works. This is primarily because understanding and appreciating a good literary work involves a considerable threshold, such as knowledge of literary theory, aesthetic sensibility, interdisciplinary knowledge. Therefore, authoritative data in this area is quite lacking. Additionally, evaluating literary works is often complex and hard to fully quantify, which directly hinders the further development of AI creation. To address this issue, this paper attempts to explore the mysteries of literary texts from the perspective of LLMs, using ancient Chinese poetry as an example for experimentation. First, we collected a variety of ancient poems from different sources and had experts annotate a small portion of them. Then, we designed a range of comprehension metrics based on LLMs to evaluate all these poems. Finally, we analyzed the correlations and differences between various poem collections to identify literary patterns. Through our experiments, we observed a series of enlightening phenomena that provide technical support for the future development of high-level literary creation based on LLMs.
Paper Structure (15 sections, 8 figures, 18 tables)

This paper contains 15 sections, 8 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Anthology Collection
  5. Metrics Calculation
  6. Pattern Summarization
  7. Discussion
  8. Perplexity
  9. Entropy
  10. Probability
  11. Embedding
  12. Frequency
  13. Model Score
  14. Future Work
  15. Conclusion

Figures (8)

  • Figure 1: Visualization of the base model early exit JSD in a poem of subpar caliber
  • Figure 2: Visualization of the sft model early exit JSD in a poem of subpar caliber
  • Figure 3: Visualization of the align model early exit JSD in a poem of subpar caliber
  • Figure 4: Visualization of the base model early exit JSD in a poem of exceptional quality
  • Figure 5: Visualization of the sft model early exit JSD in a poem of exceptional quality
  • ...and 3 more figures