Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Collaboration on Automatic Science Journalism for the General Audience

Gongyao Jiang, Xinran Shi, Qiong Luo

TL;DR

This work introduces a novel LLM-based collaboration framework (LLM-COLBR) for automatic science journalism targeted at general audiences. It deploys three agents—a journalist, a smaller reader, and a senior editor—to emulate the real-world writing-reading-feedback-revision workflow and iteratively improve article accessibility. Evaluations on SCITech, eLife, and PLOS with automatic readability metrics and human judgments show that the multi-agent collaboration yields higher readability and strong information conveyed, surpassing baselines including prompt-based and fine-tuned models, and approaching GPT-4 in authenticity. The study provides rich ablation, trend, and case analyses, proving the value of agent roles and iterative revision for accessible science communication, while noting limitations and future work on extending to multiple studies and longer contexts.

Abstract

Science journalism reports current scientific discoveries to non-specialists, aiming to enable public comprehension of the state of the art. However, this task can be challenging as the audience often lacks specific knowledge about the presented research. To address this challenge, we propose a framework that integrates three LLMs mimicking the real-world writing-reading-feedback-revision workflow, with one LLM acting as the journalist, a smaller LLM as the general public reader, and the third LLM as an editor. The journalist's writing is iteratively refined by feedback from the reader and suggestions from the editor. Our experiments demonstrate that by leveraging the collaboration of two 7B and one 1.8B open-source LLMs, we can generate articles that are more accessible than those generated by existing methods, including advanced models such as GPT-4.

LLM-Collaboration on Automatic Science Journalism for the General Audience

TL;DR

This work introduces a novel LLM-based collaboration framework (LLM-COLBR) for automatic science journalism targeted at general audiences. It deploys three agents—a journalist, a smaller reader, and a senior editor—to emulate the real-world writing-reading-feedback-revision workflow and iteratively improve article accessibility. Evaluations on SCITech, eLife, and PLOS with automatic readability metrics and human judgments show that the multi-agent collaboration yields higher readability and strong information conveyed, surpassing baselines including prompt-based and fine-tuned models, and approaching GPT-4 in authenticity. The study provides rich ablation, trend, and case analyses, proving the value of agent roles and iterative revision for accessible science communication, while noting limitations and future work on extending to multiple studies and longer contexts.

Abstract

Science journalism reports current scientific discoveries to non-specialists, aiming to enable public comprehension of the state of the art. However, this task can be challenging as the audience often lacks specific knowledge about the presented research. To address this challenge, we propose a framework that integrates three LLMs mimicking the real-world writing-reading-feedback-revision workflow, with one LLM acting as the journalist, a smaller LLM as the general public reader, and the third LLM as an editor. The journalist's writing is iteratively refined by feedback from the reader and suggestions from the editor. Our experiments demonstrate that by leveraging the collaboration of two 7B and one 1.8B open-source LLMs, we can generate articles that are more accessible than those generated by existing methods, including advanced models such as GPT-4.
Paper Structure (22 sections, 5 figures, 7 tables, 1 algorithm)

This paper contains 22 sections, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. The LLM Journalist
  4. The LLM Reader
  5. Automated Suggestions and Revisions
  6. Experiments
  7. Settings
  8. Automatic Evaluation
  9. Human Evaluation
  10. Analysis
  11. Ablation Study
  12. Trend Analysis
  13. Writing Case Analysis
  14. Case of Reading Notes
  15. Suggestions for Revision
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Reader experience varies with content technicality. Science journalism for the general audience demands high accessibility.
  • Figure 2: Overview of our LLM-Collaboration framework, LLM-CLBR.
  • Figure 3: Accessible content helps the reader take comprehensive notes.
  • Figure 4: Performance improvement over iterations, with the 0th iteration indicating the initial writing.
  • Figure 5: The questionnaire for participants to evaluate articles.