Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Group Fairness in Peer Review

Haris Aziz, Evi Micha, Nisarg Shah

TL;DR

A simple peer review model is studied, it is proved that it always admits a reviewing assignment in the core, and an efficient algorithm is designed to find one such assignment.

Abstract

Large conferences such as NeurIPS and AAAI serve as crossroads of various AI fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in a poor reviewing experience for some communities, whose submissions get assigned to less qualified reviewers outside of their communities. An often-advocated solution is to break up any such large conference into smaller conferences, but this can lead to isolation of communities and harm interdisciplinary research. We tackle this challenge by introducing a notion of group fairness, called the core, which requires that every possible community (subset of researchers) to be treated in a way that prevents them from unilaterally benefiting by withdrawing from a large conference. We study a simple peer review model, prove that it always admits a reviewing assignment in the core, and design an efficient algorithm to find one such assignment. We use real data from CVPR and ICLR conferences to compare our algorithm to existing reviewing assignment algorithms on a number of metrics.

Group Fairness in Peer Review

TL;DR

A simple peer review model is studied, it is proved that it always admits a reviewing assignment in the core, and an efficient algorithm is designed to find one such assignment.

Abstract

Large conferences such as NeurIPS and AAAI serve as crossroads of various AI fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in a poor reviewing experience for some communities, whose submissions get assigned to less qualified reviewers outside of their communities. An often-advocated solution is to break up any such large conference into smaller conferences, but this can lead to isolation of communities and harm interdisciplinary research. We tackle this challenge by introducing a notion of group fairness, called the core, which requires that every possible community (subset of researchers) to be treated in a way that prevents them from unilaterally benefiting by withdrawing from a large conference. We study a simple peer review model, prove that it always admits a reviewing assignment in the core, and design an efficient algorithm to find one such assignment. We use real data from CVPR and ICLR conferences to compare our algorithm to existing reviewing assignment algorithms on a number of metrics.
Paper Structure (14 sections, 6 theorems, 5 equations, 1 figure, 3 tables, 3 algorithms)

This paper contains 14 sections, 6 theorems, 5 equations, 1 figure, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Related Work
  4. Model
  5. CoBRA: An Algorithm for Computing Core-Based Reviewer Assignment
  6. Description of CoBRA
  7. Execution of CoBRA on a Toy Example
  8. Main Result
  9. Experiments
  10. Discussion
  11. Proof of \ref{['lem:CoBRA-validity']}
  12. Valid Assignment over the agents in $N_1$.
  13. Valid Assignment over the agents in $N_2$.
  14. Supplementary Experiments

Key Result

Theorem 1

When agent preferences are order separable and consistent, CoBRA returns an assignment in the core in $O(n^3)$ time complexity.

Figures (1)

  • Figure 1: Execution of CoBRA when $n=6$, $k_p=k_a=3$, $\sigma_{1}= 2 \succ 3 \succ 4 \succ \ldots$, $\sigma_{2}= 3 \succ 1 \succ 5 \succ \ldots$, $\sigma_3= 1 \succ 2 \succ 5 \succ \ldots$, $\sigma_4=1\succ 3 \succ 5 \succ \ldots$, $\sigma_5=6\succ 4 \succ \ldots$ and $\sigma_6=2 \succ \ldots$. On the left table, we see the assignments that are established in each round of PRA-TTC by eliminating cycles. After the execution of PRA-TTC, three papers, $p_4$, $p_5$, $p_6$ are not completely assigned. Thus, $U=\{4,5,6\}$ and $L=\{3\}$. On the right table, we see the execution of Filling-Gaps. There is a cycle in the greedy graph which is eliminated at the first round of Phase 1. In Phase 2, where $\vec{\rho}=(6,5)$, at the first round, since $p_3$ is authored by an agent in $U\cup L\setminus \{6\}$, is not reviewed by $6$ and is completely assigned, $p_3$ is assigned to $6$ while it is removed form $1$ in which $p_6$ is now assigned. At the second round, since $p_4$ is authored by an agent in $U\cup L\setminus \{5\}$, is not reviewed by $5$ and is completely assigned, $p_4$ is assigned to $5$ while it is removed form $1$ in which $p_5$ is now assigned.

Theorems & Definitions (17)

  • Definition 1: Order Separability
  • Example 1
  • Definition 2: Consistency
  • Example 2
  • Definition 3: Core
  • Theorem 1
  • proof
  • Lemma 1
  • Lemma 2
  • proof
  • ...and 7 more