Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

Sihong Wu, Yiling Ma, Yilun Zhao, Tiansheng Hu, Owen Jiang, Manasi Patwardhan, Arman Cohan

TL;DR

RbtAct is proposed, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning, and leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability.

Abstract

Large language models (LLMs) are increasingly used across the scientific workflow, including to draft peer-review reports. However, many AI-generated reviews are superficial and insufficiently actionable, leaving authors without concrete, implementable guidance and motivating the gap this work addresses. We propose RbtAct, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. Rebuttals show which reviewer comments led to concrete revisions or specific plans, and which were only defended. Building on this insight, we leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability. To support this objective, we propose a new task called perspective-conditioned segment-level review feedback generation, in which the model is required to produce a single focused comment based on the complete paper and a specified perspective such as experiments and writing. We also build a large dataset named RMR-75K that maps review segments to the rebuttal segments that address them, with perspective labels and impact categories that order author uptake. We then train the Llama-3.1-8B-Instruct model with supervised fine-tuning on review segments followed by preference optimization using rebuttal derived pairs. Experiments with human experts and LLM-as-a-judge show consistent gains in actionability and specificity over strong baselines while maintaining grounding and relevance.

RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation

TL;DR

RbtAct is proposed, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning, and leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability.

Abstract

Large language models (LLMs) are increasingly used across the scientific workflow, including to draft peer-review reports. However, many AI-generated reviews are superficial and insufficiently actionable, leaving authors without concrete, implementable guidance and motivating the gap this work addresses. We propose RbtAct, which targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. Rebuttals show which reviewer comments led to concrete revisions or specific plans, and which were only defended. Building on this insight, we leverage rebuttal as implicit supervision to directly optimize a feedback generator for actionability. To support this objective, we propose a new task called perspective-conditioned segment-level review feedback generation, in which the model is required to produce a single focused comment based on the complete paper and a specified perspective such as experiments and writing. We also build a large dataset named RMR-75K that maps review segments to the rebuttal segments that address them, with perspective labels and impact categories that order author uptake. We then train the Llama-3.1-8B-Instruct model with supervised fine-tuning on review segments followed by preference optimization using rebuttal derived pairs. Experiments with human experts and LLM-as-a-judge show consistent gains in actionability and specificity over strong baselines while maintaining grounding and relevance.
Paper Structure (70 sections, 4 equations, 13 figures, 12 tables)

This paper contains 70 sections, 4 equations, 13 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Peer Review Datasets.
  4. Review Generation.
  5. Review and Rebuttal Mapping
  6. Review–Rebuttal Source Data Collection
  7. Dataset Construction
  8. Notation.
  9. Review Segmentation.
  10. Review–Rebuttal Mapping.
  11. Cleaning and Quality Control
  12. Methodology
  13. Task Definition
  14. Review Perspective Labels.
  15. Rebuttal Impact Category as Actionability Signal.
  16. ...and 55 more sections

Figures (13)

  • Figure 1: Overview of our construction pipeline for RMR-75K.
  • Figure 2: Left: brief summary of perspective labels for review segments and impact categories for rebuttal segments. Right: normalized (100%) impact category composition by perspective.
  • Figure 3: A mapping example of our Review-Rebuttal-Mapping-75k dataset. The review and rebuttal are from the paper titled "Large Language Models as Tool Makers".
  • Figure 4: Prompt used to segment the weaknesses and questions parts of the review into segments.
  • Figure 5: Prompt used for mapping review segments with rebuttal segments.
  • ...and 8 more figures