Understanding and Supporting Peer Review Using AI-reframed Positive Summary

TL;DR

Harsh peer feedback can demotivate authors and hinder iterative improvement. The authors test AI reframing by appending AI generated positive summaries to critiques in a 2x2 online experiment varying AI framing and overall evaluation, assessing intrapersonal and interpersonal effects, revision outcomes, and motivation factors. The results show AI reframing boosts positive emotion, perceived autonomy, competence, feedback fairness, and reviewer fairness, while low overall scores increase revision quantity; seven motivational factors emerge from qualitative analysis. These findings inform design guidelines for AI assisted peer feedback that preserves author autonomy, promotes constructive and friendly exchanges, and emphasizes transparent disclosure of AI involvement to maintain trust in the review process.

Abstract

While peer review enhances writing and research quality, harsh feedback can frustrate and demotivate authors. Hence, it is essential to explore how critiques should be delivered to motivate authors and enable them to keep iterating their work. In this study, we explored the impact of appending an automatically generated positive summary to the peer reviews of a writing task, alongside varying levels of overall evaluations (high vs. low), on authors' feedback reception, revision outcomes, and motivation to revise. Through a 2x2 online experiment with 137 participants, we found that adding an AI-reframed positive summary to otherwise harsh feedback increased authors' critique acceptance, whereas low overall evaluations of their work led to increased revision efforts. We discuss the implications of using AI in peer feedback, focusing on how AI-driven critiques can influence critique acceptance and support research communities in fostering productive and friendly peer feedback practices.

Figures (6)

• Figure 1: Procedure of the controlled online experiment.
• Figure 2: Mean ($\pm$ standard error of the mean) of the perceived emotional valence of the feedback (Left; H1a) and the change in self-efficacy between initial writing and revision (Right; H1b). The X-axis shows whether participants received AI-reframed positive summary. The Y-axis of the left figure shows the perceived emotional valence. A high score indicates participants perceived positive emotion. The Y-axis of the right figure shows the change in self-efficacy. A score closer to zero indicates a smaller decrease in self-efficacy. (** indicates $p<.01$, *** indicates $p<.001$)
• Figure 3: Two bar charts with a mean ($\pm$ standard error of the mean) of participants' perceived autonomy (left) and competence (right) in making revisions. The X-axis shows the factor of the presence of AI-reframed positive summary. The Y-axis shows participants' perceived autonomy and competence. A higher score indicates a higher perceived autonomy and competence (** indicates $p<.01$).
• Figure 4: Bar charts with a mean ($\pm$ standard error of the mean) of participants' perceived usefulness (left) and fairness (right) of the feedback (A, B), and fairness and expertise of the reviewer (C, D). The X-axis shows the factor of the presence of AI-reframed positive summary. The Y-axis shows participants' perceived usefulness and fairness of the feedback. A higher score indicates higher perceived usefulness (A), fairness (B, C), and expertise (D) of the feedback and reviewers (* indicates $p<.05$, ** indicates $p<.01$).
• Figure 5: Two bar charts with a mean ($\pm$ standard error of the mean) of cosine similarity score for the revision outcome (left) and perceived effort and quality of revision (right). The X-axis shows the factor of the presence of AI-reframed positive summary. The Y-axis for the left figure shows the cosine similarity score, ranging from zero to one. A score closer to one indicates a larger difference between the initial writing and final revisions. The Y-axis for the right figure shows participants' perceived effort and quality of revision. A higher score indicates increased effort and quality for the revision (* indicates $p<.05$).