Position: The Artificial Intelligence and Machine Learning Community Should Adopt a More Transparent and Regulated Peer Review Process

TL;DR

This paper analyzes the spectrum of peer review models in AI/ML venues from fully open to fully closed, arguing for a more transparent and regulated system. Leveraging Paper Copilot, it presents open statistics and engagement metrics across 24 venues over a decade to show sustained community interest in openness. Key findings indicate that fully open reviews boost engagement and public dialogue but can affect reviewer candor, while closed reviews suffer from accountability gaps and inconsistencies in records. The authors advocate gradual moves toward open or partially open review, supported by governance, policies, and user studies, to enhance fairness, reproducibility, and global participation in AI/ML research.

Abstract

The rapid growth of submissions to top-tier Artificial Intelligence (AI) and Machine Learning (ML) conferences has prompted many venues to transition from closed to open review platforms. Some have fully embraced open peer reviews, allowing public visibility throughout the process, while others adopt hybrid approaches, such as releasing reviews only after final decisions or keeping reviews private despite using open peer review systems. In this work, we analyze the strengths and limitations of these models, highlighting the growing community interest in transparent peer review. To support this discussion, we examine insights from Paper Copilot, a website launched two years ago to aggregate and analyze AI / ML conference data while engaging a global audience. The site has attracted over 200,000 early-career researchers, particularly those aged 18-34 from 177 countries, many of whom are actively engaged in the peer review process. Drawing on our findings, this position paper advocates for a more transparent, open, and well-regulated peer review aiming to foster greater community involvement and propel advancements in the field.

Figures (9)

• Figure 1: Global 200K+ active users (K = thousands) of Paper Copilot engaging in the usage of open statistics of top AI / ML venues, distributed across 177 countries. The color scale indicates the volume of active users per country, as tracked by googleanalytics over the past two years. This distribution highlights the community's strong and widespread interest in transparency.
• Figure 2: Adoption of Review Platforms among Top-Tier AI / ML Conferences (2015–2025). Data sourced from 'Submission Instructions' or 'Call for Papers' sections on the respective venues' websites.
• Figure 3: Review Disclosure Preferences among Top-Tier AI / ML Conferences (2015–2024). Definitions of closure categories are detailed in Section \ref{['sec:analysis']}. 2025 is excluded due to some venues not having announced their preferences yet.
• Figure 4: Community Engagement on Paper Copilot: Active Users (K = thousands) visiting the Open statistics and their average engagement time (seconds). (a) Metrics by age and gender. (b) Metrics by country for the top 10 sources of traffic.
• Figure 5: Comparison of views, active users (K = thousands), average engagement time (seconds), and Google Click-Through Rate (CTR, %) across venues, categorized by actual review disclosure (fully open, partially open, fully closed). The bar values are displayed on a logarithmic scale for better visibility of differences.