Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I Will Have Order! Optimizing Orders for Fair Reviewer Assignment

Justin Payan, Yair Zick

TL;DR

The paper tackles fast, fair reviewer assignment under heterogeneous paper demands by integrating picking-sequence mechanisms with submodular optimization. It introduces GRRR, a Greedy Reviewer Round Robin algorithm that achieves a $(1+\gamma)$-approximation for welfare under approximately EF1 allocations by exploiting γ-weak submodularity and matroid constraints, and FairSequence, a fast WEf1-focused algorithm that scales to large conferences. Empirical evaluation on MIDL, CVPR, and CVPR'18 shows FairSequence is orders of magnitude faster than baselines while maintaining strong fairness guarantees, complementing GRRR’s theoretically grounded approach. The work also provides extensions for non-uniform demands and minimum reviewer supply, and includes supplementary proofs, plots, and analyses to support practical deployment in OpenReview and similar platforms.

Abstract

We present fast, fair, flexible, and welfare efficient algorithms for assigning reviewers to submitted conference papers. Our approaches extend picking sequence mechanisms, standard tools from the fair allocation literature to ensure approximate envy-freeness (typically envy-freeness up to one item, or EF1). However, fairness often comes at the cost of decreased efficiency. To overcome this challenge, we carefully select approximately optimal picking sequence orders. Applying a relaxation of submodularity, $γ$-weak submodularity, we show our Greedy Reviewer Round Robin (GRRR) approach is EF1 and yields a ${(1+γ)}$-approximation to the maximum welfare attainable by a round-robin picking sequence mechanism under any order. We present a weighted picking sequence mechanism called FairSequence that targets the Weighted EF1 criterion to offer fairness in a more general setting. Using data from three conferences, we show that FairSequence runs an order of magnitude faster and provides approximate envy-freeness guarantees that are violated by existing approaches. Its simple design also makes it very flexible to new assignment constraints. FairSequence is available in the OpenReview conference management platform, giving conference organizers access to faster reviewer assignment with high welfare and envy-freeness guarantees.

I Will Have Order! Optimizing Orders for Fair Reviewer Assignment

TL;DR

The paper tackles fast, fair reviewer assignment under heterogeneous paper demands by integrating picking-sequence mechanisms with submodular optimization. It introduces GRRR, a Greedy Reviewer Round Robin algorithm that achieves a -approximation for welfare under approximately EF1 allocations by exploiting γ-weak submodularity and matroid constraints, and FairSequence, a fast WEf1-focused algorithm that scales to large conferences. Empirical evaluation on MIDL, CVPR, and CVPR'18 shows FairSequence is orders of magnitude faster than baselines while maintaining strong fairness guarantees, complementing GRRR’s theoretically grounded approach. The work also provides extensions for non-uniform demands and minimum reviewer supply, and includes supplementary proofs, plots, and analyses to support practical deployment in OpenReview and similar platforms.

Abstract

We present fast, fair, flexible, and welfare efficient algorithms for assigning reviewers to submitted conference papers. Our approaches extend picking sequence mechanisms, standard tools from the fair allocation literature to ensure approximate envy-freeness (typically envy-freeness up to one item, or EF1). However, fairness often comes at the cost of decreased efficiency. To overcome this challenge, we carefully select approximately optimal picking sequence orders. Applying a relaxation of submodularity, -weak submodularity, we show our Greedy Reviewer Round Robin (GRRR) approach is EF1 and yields a -approximation to the maximum welfare attainable by a round-robin picking sequence mechanism under any order. We present a weighted picking sequence mechanism called FairSequence that targets the Weighted EF1 criterion to offer fairness in a more general setting. Using data from three conferences, we show that FairSequence runs an order of magnitude faster and provides approximate envy-freeness guarantees that are violated by existing approaches. Its simple design also makes it very flexible to new assignment constraints. FairSequence is available in the OpenReview conference management platform, giving conference organizers access to faster reviewer assignment with high welfare and envy-freeness guarantees.

Paper Structure

This paper contains 20 sections, 18 theorems, 8 equations, 8 figures, 7 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related work
  4. Preliminaries
  5. Fair and Efficient Assignment with Reviewer Round Robin
  6. Non-Uniform Demands and Minimum Reviewer Supply
  7. Generalized Reviewer Picking Algorithms
  8. Minimum Reviewer Supply
  9. Optimizing Orders for Picking Sequences
  10. FairSequence: a Fast and Fair Reviewer Assignment Algorithm
  11. Empirical Fairness, Efficiency, and Runtime Analysis
  12. Experimental Design
  13. Fairness and Efficiency
  14. Runtime Analysis
  15. Additional Experiments
  16. ...and 5 more sections

Key Result

Theorem 3.1

RRR terminates with an EF1 allocation where papers receive at most $k$ distinct reviewers, no reviewer $r$ is assigned to more than $u_r$ papers, and all constraints $C$ are satisfied.

Figures (8)

  • Figure 1: Envy violations and reviewer quality in CVPR, using TPMS assignment. The paper clusters with the most EF1 violations have fewer qualified reviewers relative to the cluster size.
  • Figure 2: Distribution of paper valuations for CVPR under TPMS, FairFlow, and FairSequence. FairFlow, which maximizes the minimum paper score, results in a less fair overall distribution of paper scores than FairSequence. We also show the cumulative distribution of paper scores for TPMS, FairFlow, FairSequence, and PR4A assignments. The bottom decile and quartile for FairSequence and PR4A are much higher than the bottom decile and quartile for FairFlow, and all three improve over TPMS.
  • Figure 3: Runtimes of FairSequence, TPMS, FairFlow, and PR4A on CVPR and CVPR 2018. Runtimes are not reported for MIDL since all algorithms take $<10$ seconds to run. FairSequence is at least $3$ times faster than TPMS, the second-fastest competitor.
  • Figure 4: Distribution of paper valuations for MIDL under TPMS, FairFlow, and FairSequence. We also show the cumulative distribution of paper scores for TPMS, FairFlow, FairSequence, and PR4A assignments.
  • Figure 5: Distribution of paper valuations for CVPR'18 under TPMS, FairFlow, and FairSequence. We also show the cumulative distribution of paper scores for TPMS, FairFlow, FairSequence, and PR4A assignments.
  • ...and 3 more figures

Theorems & Definitions (32)

  • Theorem 3.1
  • proof
  • Proposition 3.1
  • Theorem 4.1
  • proof
  • Theorem 4.2
  • proof
  • Theorem 4.3
  • Proposition 5.1
  • proof
  • ...and 22 more