ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing

TL;DR

This work studies how intelligent scaffolding can support novice academic peer reviewers. Through formative studies with novices and experts, the authors model expert review workflows and design ReviewFlow, an AI-assisted platform offering contextual cues, in-situ citation guidance, and notes-to-outline synthesis. In a within-subjects experiment with 16 novices, ReviewFlow yielded more comprehensive reviews and higher self-efficacy, albeit with concerns about AI biases and reliability. The findings highlight the potential and limitations of AI-driven scaffolding to improve early-career reviewers' performance without compromising community standards, and point to broader implications for knowledge-work support in science. Long-term deployment and ablation studies are proposed to refine the approach and ensure responsible use across domains.

Abstract

Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and challenges, we conducted a formative study with 10 novices and 10 experts. We found that many experts adopt a workflow of annotating, note-taking, and synthesizing notes into well-justified reviews that align with community standards. Novices lack timely guidance on how to read and assess submissions and how to structure paper reviews. To support the peer review process, we developed ReviewFlow -- an AI-driven workflow that scaffolds novices with contextual reflections to critique and annotate submissions, in-situ knowledge support to assess novelty, and notes-to-outline synthesis to help align peer reviews with community expectations. In a within-subjects experiment, 16 inexperienced reviewers wrote reviews in two conditions: using ReviewFlow and using a baseline environment with minimal guidance. With ReviewFlow, participants produced more comprehensive reviews, identifying more pros and cons. While participants appreciated the streamlined process support from ReviewFlow, they also expressed concerns about using AI as part of the scientific review process. We discuss the implications of using AI to scaffold the peer review process on scientific work and beyond.

Figures (9)

• Figure 1: ReviewFlow interface on an example paper. Users can (A) receive section-level contextual cues guided by community criteria; (B) request phrase-level contextual cues adapted to highlight paper content; (C) click the citation to get an in-situ summarization; (D) check the recommended citations not currently cited by the paper; and (E) click to summarize the notes into a high-level outline or expand into a detailed outline
• Figure 2: Experts' workflow in academic peer review, along with experts' practices, novices challenges, and design goals for each stage
• Figure 3: Contextual cues. ReviewFlow provides (A) section-level cues guided by community criteria and (B) phrase-level cues adapted to user highlighted content and selected criteria.
• Figure 4: In-situ knowledge scaffolding. ReviewFlow provides (C) a list of relevant papers from the same venue that are not cited and (D) an in-situ citation summary including title, author, and the TLDR summary.
• Figure 5: Scaffolding features support the review writing process. The top part shows Flower and Hayes cognitive process of writing flower1981cognitive. The bottom part shows the ReviewFlow features that support each writing step. On the left, ReviewFlow summarized notes into broad topics under strengths and weaknesses to facilitate planning. In the middle, use can click to expand the topics to a detailed outline. On the right, the pop up window encourages self-reflection and post-editing.