BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering

TL;DR

We address the challenge of evaluating aspect-based book summarization without manual reference summaries by introducing BookAsSumQA, a QA-driven framework that derives aspect-specific questions from a narrative knowledge graph. The method constructs a graph from chunked text, upserts entities and relations with descriptions and keywords, and generates QA pairs tied to predefined aspects; summaries are judged by how well they enable an LLM to answer these questions, using ROUGE, METEOR, and BERTScore for evaluation. Our experiments compare LLM-based and RAG-based summarization across books of varying lengths, finding that LLM-based approaches outperform on short texts while RAG-based methods gain relative strength on longer documents. This framework offers a scalable, reference-free means to assess ABS in long-form literature and points to future improvements in graph construction and retrieval strategies.

Abstract

Aspect-based summarization aims to generate summaries that highlight specific aspects of a text, enabling more personalized and targeted summaries. However, its application to books remains unexplored due to the difficulty of constructing reference summaries for long text. To address this challenge, we propose BookAsSumQA, a QA-based evaluation framework for aspect-based book summarization. BookAsSumQA automatically generates aspect-specific QA pairs from a narrative knowledge graph to evaluate summary quality based on its question-answering performance. Our experiments using BookAsSumQA revealed that while LLM-based approaches showed higher accuracy on shorter texts, RAG-based methods become more effective as document length increases, making them more efficient and practical for aspect-based book summarization.

Figures (10)

• Figure 1: In BookAsSumQA, we generate aspect-specific QA pairs from a knowledge graph and evaluate summaries by testing whether they can answer these questions, thereby assessing aspect coverage without costly human-written references.
• Figure 2: BookAsSumQA: Evaluation framework for aspect-based book summarization.
• Figure 3: QA Generation Process. (1) splitting the text into chunks and extracting entities and relations, (2) inserting the extracted entities and relations into a knowledge graph as nodes and edges, and (3) synthesizing aspect-specific QA pairs from the completed graph.
• Figure 4: (1) Hierarchical Merging and (2) Incremental Updating.
• Figure 5: The query used for RAG-ased method.