Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Streamlining the review process: AI-generated annotations in research manuscripts

Oscar Díaz, Xabier Garmendia, Juanan Pereira

TL;DR

The paper tackles the growing burden of scholarly peer review by proposing AI-assisted augmentation through LLM-generated manuscript annotations as a practical middle ground between full automation and human judgment. It introduces AnnotateGPT, a GPT-4-based browser extension that overlays color-coded, criterion-aligned excerpts to guide reviewers, using a Reverse Prompt Engineering approach to produce JSON-formatted annotations. Evaluation via a Technology Acceptance Model with nine participants indicates AnnotateGPT is perceived as useful for focus and criterion-consistency with smooth PDF workflow integration, while acknowledging limitations such as small sample size and GPT accuracy dependence. The authors discuss generalization to other stakeholders and problem spaces, advocate for larger domain-diverse studies, and highlight avenues for customizing annotation tools and refining LLM-enabled review processes for robust AI-human collaboration in scholarly publishing.

Abstract

The increasing volume of research paper submissions poses a significant challenge to the traditional academic peer-review system, leading to an overwhelming workload for reviewers. This study explores the potential of integrating Large Language Models (LLMs) into the peer-review process to enhance efficiency without compromising effectiveness. We focus on manuscript annotations, particularly excerpt highlights, as a potential area for AI-human collaboration. While LLMs excel in certain tasks like aspect coverage and informativeness, they often lack high-level analysis and critical thinking, making them unsuitable for replacing human reviewers entirely. Our approach involves using LLMs to assist with specific aspects of the review process. This paper introduces AnnotateGPT, a platform that utilizes GPT-4 for manuscript review, aiming to improve reviewers' comprehension and focus. We evaluate AnnotateGPT using a Technology Acceptance Model (TAM) questionnaire with nine participants and generalize the findings. Our work highlights annotation as a viable middle ground for AI-human collaboration in academic review, offering insights into integrating LLMs into the review process and tuning traditional annotation tools for LLM incorporation.

Streamlining the review process: AI-generated annotations in research manuscripts

TL;DR

The paper tackles the growing burden of scholarly peer review by proposing AI-assisted augmentation through LLM-generated manuscript annotations as a practical middle ground between full automation and human judgment. It introduces AnnotateGPT, a GPT-4-based browser extension that overlays color-coded, criterion-aligned excerpts to guide reviewers, using a Reverse Prompt Engineering approach to produce JSON-formatted annotations. Evaluation via a Technology Acceptance Model with nine participants indicates AnnotateGPT is perceived as useful for focus and criterion-consistency with smooth PDF workflow integration, while acknowledging limitations such as small sample size and GPT accuracy dependence. The authors discuss generalization to other stakeholders and problem spaces, advocate for larger domain-diverse studies, and highlight avenues for customizing annotation tools and refining LLM-enabled review processes for robust AI-human collaboration in scholarly publishing.

Abstract

The increasing volume of research paper submissions poses a significant challenge to the traditional academic peer-review system, leading to an overwhelming workload for reviewers. This study explores the potential of integrating Large Language Models (LLMs) into the peer-review process to enhance efficiency without compromising effectiveness. We focus on manuscript annotations, particularly excerpt highlights, as a potential area for AI-human collaboration. While LLMs excel in certain tasks like aspect coverage and informativeness, they often lack high-level analysis and critical thinking, making them unsuitable for replacing human reviewers entirely. Our approach involves using LLMs to assist with specific aspects of the review process. This paper introduces AnnotateGPT, a platform that utilizes GPT-4 for manuscript review, aiming to improve reviewers' comprehension and focus. We evaluate AnnotateGPT using a Technology Acceptance Model (TAM) questionnaire with nine participants and generalize the findings. Our work highlights annotation as a viable middle ground for AI-human collaboration in academic review, offering insights into integrating LLMs into the review process and tuning traditional annotation tools for LLM incorporation.

Paper Structure

This paper contains 15 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. The editor as the stakeholder
  4. The reviewer as the stakeholder
  5. RQ1: How could the task of LLM-authored annotations be integrated into the review process?
  6. RQ2: How could traditional annotation tools be tuned for LLM-assisted academic review?
  7. Annotation
  8. Criterion Review
  9. Full Review
  10. Evaluation
  11. Threats to validity
  12. Formalization of Learning
  13. Generalization of the Solution Instance
  14. Generalization of the Problem Instance
  15. Conclusions

Figures (5)

  • Figure 1: A review report and its UML conceptualization
  • Figure 2: Creation of annotations
  • Figure 3: Annotation-centric prompting using annotations from this very manuscript
  • Figure 4: Viewpoints for CriterionReview: the prompt includes annotations collected for the criterion at hand
  • Figure 5: Perceived Usefulness & Perceived Ease of use