Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Algorithmic Contract Design for Crowdsourced Ranking

Kiriaki Frangias, Andrew Lin, Ellen Vitercik, Manolis Zampetakis

TL;DR

This paper tackles ranking a set of items using crowdsourced pairwise comparisons from strategic agents. It introduces a contract-theory–based mechanism where the principal assigns comparisons, verifies a small subset, and pays based on verified accuracy, achieving $O(\log s)$ verifications and improved utility over solo ranking. A key technical advance is distributing comparisons via a connection to the social golfer problem, enabling per-agent workload $\tilde{O}(n^{3/2}/s)$ and exactly $r$ agents per pair, while CrowdSort identifies good agents and aggregates their inputs to recover the ground-truth ordering with high probability. Empirical results demonstrate robustness to model misspecification and show that the mechanism can outperform self-ranking under plausible disutility and reliability settings, offering scalable, incentive-compatible crowdsourced ranking for applications in search, language model feedback, and peer grading.

Abstract

Ranking is fundamental to many areas, such as search engine optimization, human feedback for language models, as well as peer grading. Crowdsourcing, which is often used for these tasks, requires proper incentivization to ensure accurate inputs. In this work, we draw on the field of \emph{contract theory} from Economics to propose a novel mechanism that enables a \emph{principal} to accurately rank a set of items by incentivizing agents to provide pairwise comparisons of the items. Our mechanism implements these incentives by verifying a subset of each agent's comparisons, a task we assume to be costly. The agent is compensated (for example, monetarily or with class credit) based on the accuracy of these comparisons. Our mechanism achieves the following guarantees: (1) it only requires the principal to verify $O(\log s)$ comparisons, where $s$ is the total number of agents, and (2) it provably achieves higher total utility for the principal compared to ranking the items herself with no crowdsourcing.

Algorithmic Contract Design for Crowdsourced Ranking

TL;DR

This paper tackles ranking a set of items using crowdsourced pairwise comparisons from strategic agents. It introduces a contract-theory–based mechanism where the principal assigns comparisons, verifies a small subset, and pays based on verified accuracy, achieving verifications and improved utility over solo ranking. A key technical advance is distributing comparisons via a connection to the social golfer problem, enabling per-agent workload and exactly agents per pair, while CrowdSort identifies good agents and aggregates their inputs to recover the ground-truth ordering with high probability. Empirical results demonstrate robustness to model misspecification and show that the mechanism can outperform self-ranking under plausible disutility and reliability settings, offering scalable, incentive-compatible crowdsourced ranking for applications in search, language model feedback, and peer grading.

Abstract

Ranking is fundamental to many areas, such as search engine optimization, human feedback for language models, as well as peer grading. Crowdsourcing, which is often used for these tasks, requires proper incentivization to ensure accurate inputs. In this work, we draw on the field of \emph{contract theory} from Economics to propose a novel mechanism that enables a \emph{principal} to accurately rank a set of items by incentivizing agents to provide pairwise comparisons of the items. Our mechanism implements these incentives by verifying a subset of each agent's comparisons, a task we assume to be costly. The agent is compensated (for example, monetarily or with class credit) based on the accuracy of these comparisons. Our mechanism achieves the following guarantees: (1) it only requires the principal to verify comparisons, where is the total number of agents, and (2) it provably achieves higher total utility for the principal compared to ranking the items herself with no crowdsourcing.
Paper Structure (33 sections, 23 theorems, 57 equations, 10 figures, 1 table, 4 algorithms)

This paper contains 33 sections, 23 theorems, 57 equations, 10 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Key challenges.
  3. Our contributions
  4. Related work
  5. Rank Aggregation.
  6. Crowdsourcing.
  7. Ordinal peer grading.
  8. Warm-up: Known and equal disutilities
  9. Model and notation
  10. Agents' incentives
  11. Peer grading algorithm
  12. Allocation of items
  13. Assigning verified comparisons.
  14. Assigning unverified comparisons.
  15. Principal's incentives
  16. ...and 18 more sections

Key Result

Lemma 2.2

If the principal pays each agent $p^* \geq \frac{d\psi}{\hat{\pi}\pi},$ then every agent is incentivized to exert effort.

Figures (10)

  • Figure 1: Principal's utility as a function of $\psi$.
  • Figure 2: Plots showing the number of comparisons returned as a function of $\pi$ and $\psi$. The heat plots indicate the support of uniformly sampled noise added to $\psi$ and $\pi$.
  • Figure 3: Principal's utility as a function of the number of agents incentivized $g$.
  • Figure 4: Principal's utility as a function of parameter $\pi$.
  • Figure 5: Principal's utility as a function of parameter $\psi$ (in the absence of noise)
  • ...and 5 more figures

Theorems & Definitions (41)

  • Remark 2.1
  • Remark 2.2
  • Lemma 2.2
  • Definition 2.3
  • Lemma 2.3
  • proof : Proof sketch
  • Lemma 2.3
  • Lemma 2.3
  • Lemma 2.3
  • Theorem 2.4
  • ...and 31 more