Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approach

Javier Coronado-Blázquez

TL;DR

This paper investigates whether large language models can generate accurate, detailed book summaries from internal knowledge alone, without access to the original texts. It combines a cross-model generation framework with an LLM-as-a-judge evaluation paradigm, using ROUGE and BERTScore alongside model-based rationales to assess content fidelity and semantic similarity. The study reveals substantial variability across books and models, limited alignment between automatic similarity metrics and human-like judgments, and hints at biases in self-evaluation. These findings highlight the challenges of relying on internal encodings for factual summaries and underscore the need for robust, multi-faceted evaluation pipelines to drive improvements in factuality and consistency of AI-generated narratives.

Abstract

We study the ability of large language models (LLMs) to generate comprehensive and accurate book summaries solely from their internal knowledge, without recourse to the original text. Employing a diverse set of books and multiple LLM architectures, we examine whether these models can synthesize meaningful narratives that align with established human interpretations. Evaluation is performed with a LLM-as-a-judge paradigm: each AI-generated summary is compared against a high-quality, human-written summary via a cross-model assessment, where all participating LLMs evaluate not only their own outputs but also those produced by others. This methodology enables the identification of potential biases, such as the proclivity for models to favor their own summarization style over others. In addition, alignment between the human-crafted and LLM-generated summaries is quantified using ROUGE and BERTScore metrics, assessing the depth of grammatical and semantic correspondence. The results reveal nuanced variations in content representation and stylistic preferences among the models, highlighting both strengths and limitations inherent in relying on internal knowledge for summarization tasks. These findings contribute to a deeper understanding of LLM internal encodings of factual information and the dynamics of cross-model evaluation, with implications for the development of more robust natural language generative systems.

Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approach

TL;DR

This paper investigates whether large language models can generate accurate, detailed book summaries from internal knowledge alone, without access to the original texts. It combines a cross-model generation framework with an LLM-as-a-judge evaluation paradigm, using ROUGE and BERTScore alongside model-based rationales to assess content fidelity and semantic similarity. The study reveals substantial variability across books and models, limited alignment between automatic similarity metrics and human-like judgments, and hints at biases in self-evaluation. These findings highlight the challenges of relying on internal encodings for factual summaries and underscore the need for robust, multi-faceted evaluation pipelines to drive improvements in factuality and consistency of AI-generated narratives.

Abstract

We study the ability of large language models (LLMs) to generate comprehensive and accurate book summaries solely from their internal knowledge, without recourse to the original text. Employing a diverse set of books and multiple LLM architectures, we examine whether these models can synthesize meaningful narratives that align with established human interpretations. Evaluation is performed with a LLM-as-a-judge paradigm: each AI-generated summary is compared against a high-quality, human-written summary via a cross-model assessment, where all participating LLMs evaluate not only their own outputs but also those produced by others. This methodology enables the identification of potential biases, such as the proclivity for models to favor their own summarization style over others. In addition, alignment between the human-crafted and LLM-generated summaries is quantified using ROUGE and BERTScore metrics, assessing the depth of grammatical and semantic correspondence. The results reveal nuanced variations in content representation and stylistic preferences among the models, highlighting both strengths and limitations inherent in relying on internal knowledge for summarization tasks. These findings contribute to a deeper understanding of LLM internal encodings of factual information and the dynamics of cross-model evaluation, with implications for the development of more robust natural language generative systems.

Paper Structure

This paper contains 6 sections, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Results
  4. Conclusions
  5. Book selection
  6. Examples of LLM-as-a-judge score justifications

Figures (6)

  • Figure 1: Heatmap of the ROUGE (ROUGE-1, ROUGE-2 and ROUGE-L) and BERTScore (F1) metrics, aggregated for the different LLMs.
  • Figure 2: Comparison of ROUGE-1 and BERTScore metrics across the five calls per book and model. Error bars represent the 68% confidence level. From top to bottom: results for GPT-4o-mini, Mistral-7b and DeepSeek-R1
  • Figure 3: Distribution of LLM-based scores for all the books studied, in alphabetical order of title. Each histogram represents the distribution of all ratings in every combination of generator+judge. In the upper left corner of each subplot, the mean $\pm$ standard deviation of the distribution is shown.
  • Figure 4: Heatmap of scores on all combinations of generator+judge, averaged for all books in the sample. Diagonal cells represent self-evaluated models.
  • Figure 5: Heatmap of scores on all combinations of generator+judge, for three specific books. Top panel: To Kill a Mocking Bird. Middle panel: The Bell Jar. Bottom panel: The Man in the High Castle.
  • ...and 1 more figures