A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions

Abstract

Peer review often involves reviewers submitting their independent reviews, followed by a discussion among reviewers of each paper. A question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this by conducting a randomized controlled trial at the UAI 2022 conference. We randomly split the reviewers and papers into two conditions--one with anonymous discussions and the other with non-anonymous discussions, and conduct an anonymous survey of all reviewers, to address the following questions: 1. Do reviewers discuss more in one of the conditions? Marginally more in anonymous (n = 2281, p = 0.051). 2. Does seniority have more influence on final decisions when non-anonymous? Yes, the decisions are closer to senior reviewers' scores in the non-anonymous condition than in anonymous (n = 484, p = 0.04). 3. Are reviewers more polite in one of the conditions? No significant difference in politeness of reviewers' text-based responses (n = 1125, p = 0.72). 4. Do reviewers' self-reported experiences differ across the two conditions? No significant difference for each of the five questions asked (n = 132 and p > 0.3). 5. Do reviewers prefer one condition over the other? Yes, there is a weak preference for anonymous discussions (n = 159 and Cohen's d= 0.25). 6. What do reviewers consider important to make policy on anonymity among reviewers? Reviewers' feeling of safety in expressing their opinions was rated most important, while polite communication among reviewers was rated least important (n = 159). 7. Have reviewers experienced dishonest behavior due to non-anonymity in discussions? Yes, roughly 7% of respondents answered affirmatively (n = 167). Overall, this experiment reveals evidence supporting an anonymous discussion setup in the peer-review process, in terms of the evaluation criteria considered.

Figures (4)

• Figure 1: Visualization of the distribution of count of discussion posts written by a reviewer for a paper in the anonymous and non-anonymous condition in UAI 2022. The x-axis shows the number of posts made for by a reviewer for a paper.
• Figure 2: Visualization of the distribution of politeness scores obtained for all the discussion posts written by reviewers in the anonymous and non-anonymous condition in UAI 2022. The height of the left-most bar indicates the sample probability of a score falling in the interval [1,2) and so on for each bar.
• Figure 3: Survey outcomes on reviewers' self-reported experience in the two conditions. Respondents provided Likert-style responses for each question which have been visualized here. The markers 'NA' and 'A' indicate responses from the non-anonymous and the anonymous condition respectively. For each survey question, we tested for difference in the participants' responses across the two conditions employing the Mann-Whitney U test. There was no significant difference in participants' experience corresponding to any of the survey questions. More details are provided in Table \ref{['table:survey_results']}.
• Figure 4: Survey outcomes on reviewers' ranking of importance of different aspects in making conference policy decisions regarding anonymity between reviewers in discussions.