Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ARC: Argument Representation and Coverage Analysis for Zero-Shot Long Document Summarization with Instruction Following LLMs

Mohamed Elaraby, Diane Litman

TL;DR

This paper introduces Argument Representation Coverage (ARC), a multi-granularity evaluation framework for assessing how well instruction-tuned LLMs preserve salient argumentative content in long-form summaries. ARC defines fullset, role, and atomic coverage using a Phi function and evaluates summaries across CANLII and DRI corpora, employing GPT-4o judgments and classifiers to approximate large-scale assessments. Findings show that ARC atomic coverage correlates most strongly with expert judgments, but LLMs exhibit notable positional and role biases, with sparse legal arguments being particularly challenging to preserve. The work highlights the need for argument-aware summarization strategies and explicit argument representations to improve reliability in high-stakes domains.

Abstract

Integrating structured information has long improved the quality of abstractive summarization, particularly in retaining salient content. In this work, we focus on a specific form of structure: argument roles, which are crucial for summarizing documents in high-stakes domains such as law. We investigate whether instruction-tuned large language models (LLMs) adequately preserve this information. To this end, we introduce Argument Representation Coverage (ARC), a framework for measuring how well LLM-generated summaries capture salient arguments. Using ARC, we analyze summaries produced by three open-weight LLMs in two domains where argument roles are central: long legal opinions and scientific articles. Our results show that while LLMs cover salient argument roles to some extent, critical information is often omitted in generated summaries, particularly when arguments are sparsely distributed throughout the input. Further, we use ARC to uncover behavioral patterns -- specifically, how the positional bias of LLM context windows and role-specific preferences impact the coverage of key arguments in generated summaries, emphasizing the need for more argument-aware summarization strategies.

ARC: Argument Representation and Coverage Analysis for Zero-Shot Long Document Summarization with Instruction Following LLMs

TL;DR

This paper introduces Argument Representation Coverage (ARC), a multi-granularity evaluation framework for assessing how well instruction-tuned LLMs preserve salient argumentative content in long-form summaries. ARC defines fullset, role, and atomic coverage using a Phi function and evaluates summaries across CANLII and DRI corpora, employing GPT-4o judgments and classifiers to approximate large-scale assessments. Findings show that ARC atomic coverage correlates most strongly with expert judgments, but LLMs exhibit notable positional and role biases, with sparse legal arguments being particularly challenging to preserve. The work highlights the need for argument-aware summarization strategies and explicit argument representations to improve reliability in high-stakes domains.

Abstract

Integrating structured information has long improved the quality of abstractive summarization, particularly in retaining salient content. In this work, we focus on a specific form of structure: argument roles, which are crucial for summarizing documents in high-stakes domains such as law. We investigate whether instruction-tuned large language models (LLMs) adequately preserve this information. To this end, we introduce Argument Representation Coverage (ARC), a framework for measuring how well LLM-generated summaries capture salient arguments. Using ARC, we analyze summaries produced by three open-weight LLMs in two domains where argument roles are central: long legal opinions and scientific articles. Our results show that while LLMs cover salient argument roles to some extent, critical information is often omitted in generated summaries, particularly when arguments are sparsely distributed throughout the input. Further, we use ARC to uncover behavioral patterns -- specifically, how the positional bias of LLM context windows and role-specific preferences impact the coverage of key arguments in generated summaries, emphasizing the need for more argument-aware summarization strategies.

Paper Structure

This paper contains 36 sections, 3 equations, 10 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Information Saliency in LLMs.
  4. Limitations of LLMs in Long-Document Summarization.
  5. Argument Mining and Abstractive Summarization.
  6. Evaluation Metrics for Long-Form Summarization.
  7. Datasets
  8. Legal Opinions: CANLII
  9. Scientific Articles: DRI
  10. The ARC framework
  11. Overview and Notations
  12. Full Argument Set Coverage
  13. Independent Argument Role Coverage
  14. Subatomic Argument Coverage
  15. Choice of $\Phi$
  16. ...and 21 more sections

Figures (10)

  • Figure 1: Examples of ARC scores at multiple granularities—fullset, role, and atomic (where SAF denotes subatomic facts)—for a summary generated by LlaMA3.18B on a case from the long-legal opinions dataset.
  • Figure 2: Average ARC$_{\text{atomic}}$ across CANLII and DRI datasets.
  • Figure 3: Error types distribution of $\texttt{ARC}_{\text{atomic}}$.
  • Figure 4: Source sentences relative position in the LLM context window across all models and various argument roles for both CANLII and DRI corpora.
  • Figure 5: Bias score $\beta$ across multiple argument roles (controlled length and non-controlled length) for both CANLII and DRI corpora.
  • ...and 5 more figures