On the Detection of Reviewer-Author Collusion Rings From Paper Bidding

TL;DR

This work tackles the problem of detecting reviewer-author collusion rings from paper bidding data in computer science conferences. It formalizes two graph representations (unipartite reviewer graph and bipartite reviewer-paper graph), injects collusion behavior, and evaluates multiple dense-subgraph detection algorithms. The key finding is that colluders can meaningfully influence paper assignments while remaining largely undetected by bid-based detectors, with average overlap with true colluders often limited (e.g., around 0.31 in challenging settings). The results highlight the need for richer metadata and more robust detection methods beyond bidding alone, informing future research on collusion mitigation and detection in peer review.

Abstract

A major threat to the peer-review systems of computer science conferences is the existence of "collusion rings" between reviewers. In such collusion rings, reviewers who have also submitted their own papers to the conference work together to manipulate the conference's paper assignment, with the aim of being assigned to review each other's papers. The most straightforward way that colluding reviewers can manipulate the paper assignment is by indicating their interest in each other's papers through strategic paper bidding. One potential approach to solve this important problem would be to detect the colluding reviewers from their manipulated bids, after which the conference can take appropriate action. While prior work has developed effective techniques to detect other kinds of fraud, no research has yet established that detecting collusion rings is even possible. In this work, we tackle the question of whether it is feasible to detect collusion rings from the paper bidding. To answer this question, we conduct empirical analysis of two realistic conference bidding datasets, including evaluations of existing algorithms for fraud detection in other applications. We find that collusion rings can achieve considerable success at manipulating the paper assignment while remaining hidden from detection: for example, in one dataset, undetected colluders are able to achieve assignment to up to 30% of the papers authored by other colluders. In addition, when 10 colluders bid on all of each other's papers, no detection algorithm outputs a group of reviewers with more than 31% overlap with the true colluders. These results suggest that collusion cannot be effectively detected from the bidding using popular existing tools, demonstrating the need to develop more complex detection algorithms as well as those that leverage additional metadata (e.g., reviewer-paper text-similarity scores).

Figures (10)

• Figure 1: Exact counts of honest-reviewer groups with varying size and edge density (Figures \ref{['fig:exact_aamas']}-\ref{['fig:exact_wu']}), and the size and edge density of honest-reviewer groups found by a heuristic method (Figures \ref{['fig:frontier_aamas']}-\ref{['fig:frontier_wu']}). In Figures \ref{['fig:exact_aamas']}-\ref{['fig:exact_wu']}, values in cells marked with "$\geq$" represent lower bounds since exact counts were infeasible to compute. In Figures \ref{['fig:frontier_aamas']}-\ref{['fig:frontier_wu']}, each point corresponds to an existing honest-reviewer group (found by a greedy peeling method), and the shaded area indicates the region in which there exists at least one honest-reviewer group. Note that the vertical axis does not start at 0 for easier comparison with other figures.
• Figure 2: Performance of detection algorithms on $\mathcal{G}_{1}$. Values indicate the mean Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors. Higher values correspond to better detection performance.
• Figure 3: Success of colluders in terms of $k$ and $\gamma$. Values indicate the mean for each metric along with standard errors.
• Figure 4: Exact counts of honest-reviewer groups with varying size and bid density.
• Figure 5: Performance of detection algorithms on $\mathcal{G}_{2}$. Values indicate the mean Jaccard similarity between the true set of colluders and the algorithm output, along with standard errors. Higher values correspond to better detection performance.