JRE-L: Journalist, Reader, and Editor LLMs in the Loop for Science Journalism for the General Audience

TL;DR

The paper introduces JRE-L, a parameter-free, three-LLM framework for automatic science journalism that integrates a journalist LLM, a general-reader LLM, and an editor LLM in an iterative writing-reading-feedback-revision loop. Through five iterations, the journalist’s article is progressively refined based on reader notes and editor feedback, resulting in higher readability and competitive information conveyance compared to single-LLM prompting and fine-tuning baselines. The approach is evaluated on SCITech, eLife, and PLOS datasets using automatic readability metrics (CLI, FKGL, DCRS) and human judgments across readability, information conveyance, authenticity, and interestingness, with JRE-L generally outperforming baselines and approaching GPT-4 in quality while using smaller open-source models. Ablation studies confirm that each component—reader, editor, and collaboration—contributes to performance, and case analyses demonstrate clearer narratives and better explanations of technical terms. The work offers a practical, scalable pathway for ASJ by leveraging collaborative LLMs to tailor scientific content for a general audience, and it discusses limitations and future work, including multi-paper summaries, semantic evaluation, longer contexts, and human-preference optimization.

Abstract

Science journalism reports current scientific discoveries to non-specialists, aiming to enable public comprehension of the state of the art. This task is challenging as the audience often lacks specific knowledge about the presented research. We propose a JRE-L framework that integrates three LLMs mimicking the writing-reading-feedback-revision loop. In JRE-L, one LLM acts as the journalist, another LLM as the general public reader, and the third LLM as an editor. The journalist's writing is iteratively refined by feedback from the reader and suggestions from the editor. Our experiments demonstrate that by leveraging the collaboration of two 7B and one 1.8B open-source LLMs, we can generate articles that are more accessible than those generated by existing methods, including prompting single advanced models such as GPT-4 and other LLM-collaboration strategies. Our code is publicly available at github.com/Zzoay/JRE-L.

Figures (5)

• Figure 1: An article written by a science journalist may be challenging for the general reader without the reader's feedback to the editor in the revision cycle (a). Incorporating the reader's feedback into the journalism cycle can help enhance the readability of the article (b).
• Figure 2: Overview of our iterative ASJ framework, JRE-L.
• Figure 3: Accessible content helps the reader take comprehensive notes.
• Figure 4: Performance improvement over iterations, with iteration 0 producing the initial writing.
• Figure 5: The questionnaire for participants to evaluate articles.