Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Surprising Effectiveness of SP Voting with Partial Preferences

Hadi Hosseini, Debmalya Mandal, Amrit Puhan

TL;DR

This paper tackles recovering ground-truth rankings over a large set of alternatives when the majority may be misinformed. It extends the surprisingly popular (SP) voting framework to partial preferences by introducing Partial-SP and Aggregated-SP, which operate on small subsets $T$ of size $k\ll m$ and leverage various elicitation formats. Through a large MTurk study and a concentric Mallows-model simulation, the authors show that SP-based aggregation with partial information yields superior recovery of the full ranking compared to classical methods, and they provide theoretical sample-complexity bounds showing how performance scales with $k$ and the fraction of experts. The work offers scalable, information-efficient tools for crowd-based ranking in high-dimensional settings and suggests directions for future work on more general probabilistic models and robustness to misinformation.

Abstract

We consider the problem of recovering the ground truth ordering (ranking, top-$k$, or others) over a large number of alternatives. The wisdom of crowd is a heuristic approach based on Condorcet's Jury theorem to address this problem through collective opinions. This approach fails to recover the ground truth when the majority of the crowd is misinformed. The surprisingly popular (SP) algorithm cite{prelec2017solution} is an alternative approach that is able to recover the ground truth even when experts are in minority. The SP algorithm requires the voters to predict other voters' report in the form of a full probability distribution over all rankings of alternatives. However, when the number of alternatives, $m$, is large, eliciting the prediction report or even the vote over $m$ alternatives might be too costly. In this paper, we design a scalable alternative of the SP algorithm which only requires eliciting partial preferences from the voters, and propose new variants of the SP algorithm. In particular, we propose two versions -- Aggregated-SP and Partial-SP -- that ask voters to report vote and prediction on a subset of size $k$ ($\ll m$) in terms of top alternative, partial rank, or an approval set. Through a large-scale crowdsourcing experiment on MTurk, we show that both of our approaches outperform conventional preference aggregation algorithms for the recovery of ground truth rankings, when measured in terms of Kendall-Tau distance and Spearman's $ρ$. We further analyze the collected data and demonstrate that voters' behavior in the experiment, including the minority of the experts, and the SP phenomenon, can be correctly simulated by a concentric mixtures of Mallows model. Finally, we provide theoretical bounds on the sample complexity of SP algorithms with partial rankings to demonstrate the theoretical guarantees of the proposed methods.

The Surprising Effectiveness of SP Voting with Partial Preferences

TL;DR

This paper tackles recovering ground-truth rankings over a large set of alternatives when the majority may be misinformed. It extends the surprisingly popular (SP) voting framework to partial preferences by introducing Partial-SP and Aggregated-SP, which operate on small subsets of size and leverage various elicitation formats. Through a large MTurk study and a concentric Mallows-model simulation, the authors show that SP-based aggregation with partial information yields superior recovery of the full ranking compared to classical methods, and they provide theoretical sample-complexity bounds showing how performance scales with and the fraction of experts. The work offers scalable, information-efficient tools for crowd-based ranking in high-dimensional settings and suggests directions for future work on more general probabilistic models and robustness to misinformation.

Abstract

We consider the problem of recovering the ground truth ordering (ranking, top-, or others) over a large number of alternatives. The wisdom of crowd is a heuristic approach based on Condorcet's Jury theorem to address this problem through collective opinions. This approach fails to recover the ground truth when the majority of the crowd is misinformed. The surprisingly popular (SP) algorithm cite{prelec2017solution} is an alternative approach that is able to recover the ground truth even when experts are in minority. The SP algorithm requires the voters to predict other voters' report in the form of a full probability distribution over all rankings of alternatives. However, when the number of alternatives, , is large, eliciting the prediction report or even the vote over alternatives might be too costly. In this paper, we design a scalable alternative of the SP algorithm which only requires eliciting partial preferences from the voters, and propose new variants of the SP algorithm. In particular, we propose two versions -- Aggregated-SP and Partial-SP -- that ask voters to report vote and prediction on a subset of size () in terms of top alternative, partial rank, or an approval set. Through a large-scale crowdsourcing experiment on MTurk, we show that both of our approaches outperform conventional preference aggregation algorithms for the recovery of ground truth rankings, when measured in terms of Kendall-Tau distance and Spearman's . We further analyze the collected data and demonstrate that voters' behavior in the experiment, including the minority of the experts, and the SP phenomenon, can be correctly simulated by a concentric mixtures of Mallows model. Finally, we provide theoretical bounds on the sample complexity of SP algorithms with partial rankings to demonstrate the theoretical guarantees of the proposed methods.
Paper Structure (42 sections, 4 theorems, 50 equations, 23 figures, 5 tables, 3 algorithms)

This paper contains 42 sections, 4 theorems, 50 equations, 23 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model
  4. Elicitation Formats
  5. Aggregation Algorithms for Partial Preferences
  6. Experimental Design
  7. Results and Analysis
  8. Accuracy Metrics
  9. Predicting the Ground Truth Ranking
  10. Simulated Model of Voter Behavior
  11. Analysis of Sample Complexity
  12. Discussion and Future Work
  13. Appendix
  14. Formalism of Elicitation Formats
  15. Common Voting Rules
  16. ...and 27 more sections

Key Result

Theorem 1

Suppose asn:identifiability holds, and the total number of samples $n \ge k! \sqrt{\frac{10 k \log(2k/\delta)}{ \mu}}$ where $\mu = p \cdot \frac{Z(\phi_E, m-k)}{Z(\phi_E)} \cdot \phi_E^{k(k-1)/2}+ (1-p) \cdot \frac{Z(\phi_{NE}, m-k)}{Z(\phi_{NE})}\cdot \phi_{NE}^{k(k-1)/2}$. Then the surprisingly p

Figures (23)

  • Figure 1: Workflow of a participant
  • Figure 2: Comparing the predicted and ground-truth rankings for different elicitation formats using Kendall-Tau and Spearman's $\rho$ correlations (higher is better). All results use Copeland as their aggregation rule.
  • Figure 3: Comparing the Partial-SP algorithm with Copeland (no prediction information) measured by pairwise and Top-$t$ hit rates. The elicitation format is Approval(2)-Approval(2).
  • Figure 4: Comparing the predicted and ground-truth rankings for different aggregation rules using Kendall-Tau and Spearman's $\rho$ correlations (higher is better). The elicitation format is Rank-Rank; each comparison uses the same aggregation rule in the SP algorithm.
  • Figure 5: Comparison of inferred parameters of the Concentric mixtures of Mallows model for real and synthetic data. The experts vote closer to and predict farther from the ground-truth. The non-experts vote and predict far from the ground truth. The proportion of experts in both datasets was found to be less than 20%.
  • ...and 18 more figures

Theorems & Definitions (10)

  • Theorem 1
  • Corollary 1
  • Example 1
  • Example 2
  • Example 3
  • Example 4
  • proof
  • Lemma 1: Theorem 9 of canonne2020short
  • Lemma 2
  • proof