AgentReview: Exploring Peer Review Dynamics with LLM Agents

TL;DR

This work introduces AgentReview, an LLM-based peer-review simulation framework that uses agent-based modeling to disentangle multivariate and latent factors in the review process while protecting reviewer privacy. By integrating GPT-4/Llama with AutoGPT and simulating four years of ICLR submissions, the study demonstrates how reviewer commitment, intention, knowledgeability, area-chair involvement, author anonymity, and review-mechanism design shape outcomes. Key findings include a 37.1% variation in decisions due to biases, notable conformity and halo effects, and substantial influence of AC styles and author identities on final judgments. The framework provides a versatile, privacy-preserving testbed for evaluating and designing fairer, more transparent peer-review mechanisms, with a large synthetic dataset to support future research.

Abstract

Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms. Our code is available at https://github.com/Ahren09/AgentReview.

Figures (10)

• Figure 1: AgentReview is an open and flexible framework designed to realistically simulate the peer review process. It enables controlled experiments to disentangle multiple variables in peer review, allowing for an in-depth examination of their effects on review outcomes. Our findings align with established sociological theories.
• Figure 2: Our paper review pipeline consists of 5 phases. Solid black arrows $\rightarrow$ represent authorship connections, while blue dashed arrow $\rightarrow$ indicate visibility relations.
• Figure 3: Distribution of initial and final scores with respect to varying number of irresponsible (left) & malicious (right) reviewers.
• Figure 4: Distribution of reasons for acceptance and rejections.
• Figure 5: Similarities between reviews and meta-reviews w/ various intervention strategies from AC. Left: BERTScore, right: sentence embedding similarity.