Re$^2$: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions

TL;DR

Re^2 addresses data diversity and consistency gaps in peer-review data by building a large, consistency-ensured OpenReview-derived dataset that includes initial submissions, reviews, and structured rebuttal dialogues across 24 conferences and 21 workshops. It comprises two parts: model-Review for static evaluation tasks and model-Rebuttal for multi-turn rebuttal conversations, enabling both traditional analysis and interactive, dialogue-based training of LLMs. Empirical results across acceptance/score prediction, review generation, and rebuttal-discussion show finetuned models achieving strong performance, validating the dataset's utility for advancing automated peer-review tools. This resource has practical significance in reducing submission burdens and improving review quality, with future work including multimodal (vision-language) extensions.

Abstract

Peer review is a critical component of scientific progress in the fields like AI, but the rapid increase in submission volume has strained the reviewing system, which inevitably leads to reviewer shortages and declines review quality. Besides the growing research popularity, another key factor in this overload is the repeated resubmission of substandard manuscripts, largely due to the lack of effective tools for authors to self-evaluate their work before submission. Large Language Models (LLMs) show great promise in assisting both authors and reviewers, and their performance is fundamentally limited by the quality of the peer review data. However, existing peer review datasets face three major limitations: (1) limited data diversity, (2) inconsistent and low-quality data due to the use of revised rather than initial submissions, and (3) insufficient support for tasks involving rebuttal and reviewer-author interactions. To address these challenges, we introduce the largest consistency-ensured peer review and rebuttal dataset named Re^2, which comprises 19,926 initial submissions, 70,668 review comments, and 53,818 rebuttals from 24 conferences and 21 workshops on OpenReview. Moreover, the rebuttal and discussion stage is framed as a multi-turn conversation paradigm to support both traditional static review tasks and dynamic interactive LLM assistants, providing more practical guidance for authors to refine their manuscripts and helping alleviate the growing review burden. Our data and code are available in https://anonymous.4open.science/r/ReviewBench_anon/.

Figures (3)

• Figure 1: The conversion from raw review and rebuttal data to multi-turn conversations. For the raw review data crawled from OpenReview (as shown in sub-figure(a)), we concatenate multiple consecutive responses from the same role (author or reviewer) into a single turn. In cases where the author's final response is merely a reminder or urging, we adopt a hybrid strategy combining manual inspection and automated methods to identify and remove such reminder responses. As for the global responses, we insert them into the dialogue at the appropriate position, treating it as supplementary reference rather than direct conversation content. Finally, as shown in sub-figure(b), we construct a self-consistent, high-quality, and information-complete multi-turn conversation dataset.
• Figure 2: Statistics of our model dataset. (a) Distribution of the length of papers in tokens. (b) Distribution of the length of reviews in tokens. (c) Distribution of the number of papers and reviews in each conference. (d) Distribution of the number of rebuttals in each conference of each year. (e) Submission counts and acceptance proportion across the 10 most frequent keywords. (f) Violin plot (with a box plot inside) of review scores across the top 10 conferences with the most papers.
• Figure 3: Details of the Violin Plot. The upper and lower whiskers represent the maximum and minimum observed values. The third quartile ($\text{Q}_3$) is the value below which 75% of the data falls. The first quartile ($\text{Q}_1$) is the value below which 25% of the data falls. IQR represents the distribution of the middle 50% of the data. The tail line can extend beyond the data boundaries, reflecting the smoothness of the curve. The median line's position varies with the data distribution. The contour width represents the density of the number of papers.