Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

Marcio Fonseca, Shay B. Cohen

TL;DR

The paper investigates how large language model (LLM) summarizers can be steered toward diverse scientific communication goals without fine-tuning. It formalizes controllable summarization via prompts that specify conciseness, narrative perspective, and keyword coverage, and employs classifier-free guidance to bias outputs toward these intents. Across MuP reviewer summaries, arXiv/PubMed abstracts, and eLife lay summaries, LLMs achieve competitive lexical overlap and are frequently preferred by humans, especially for shorter, goal-directed tasks; however, generating long, highly abstractive lay summaries remains challenging. The findings highlight both the potential and limits of instruction-based control for domain-specific summarization and advocate for evaluation frameworks that account for communicative intent rather than relying solely on reference-based metrics.

Abstract

In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.

Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

TL;DR

The paper investigates how large language model (LLM) summarizers can be steered toward diverse scientific communication goals without fine-tuning. It formalizes controllable summarization via prompts that specify conciseness, narrative perspective, and keyword coverage, and employs classifier-free guidance to bias outputs toward these intents. Across MuP reviewer summaries, arXiv/PubMed abstracts, and eLife lay summaries, LLMs achieve competitive lexical overlap and are frequently preferred by humans, especially for shorter, goal-directed tasks; however, generating long, highly abstractive lay summaries remains challenging. The findings highlight both the potential and limits of instruction-based control for domain-specific summarization and advocate for evaluation frameworks that account for communicative intent rather than relying solely on reference-based metrics.

Abstract

In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.
Paper Structure (45 sections, 5 equations, 2 figures, 17 tables)

This paper contains 45 sections, 5 equations, 2 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Guiding Summarizer Intentions
  3. Intention Prompt Templates
  4. Style Intention Prompts
  5. Conciseness prompt
  6. Narrative perspective prompt
  7. Keyword Coverage Prompt
  8. Keyword model
  9. Intention Control Metrics
  10. Conciseness
  11. Narrative perspective
  12. Keyword Coverage
  13. Experiments
  14. Datasets
  15. Multi-perspective scientific summarization
  16. ...and 30 more sections

Figures (2)

  • Figure 1: An overview of our controllability experiments. We expose LLM summarizers to prompts conveying communicative intentions related to conciseness, narrative perspective, and keywords inferred by a keyword model. Then, we measure how generated summaries adhere to those intentional targets.
  • Figure 2: Number of sentences in generated summaries subject to varying conciseness targets (100 samples from eLife validation set).