Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aggregating Quantitative Relative Judgments: From Social Choice to Ranking Prediction

Yixuan Even Xu, Hanrui Zhang, Yu Cheng, Vincent Conitzer

TL;DR

This work introduces new aggregation rules for QRJA and study their structural and computational properties, and evaluates the proposed methods on data from various real races and shows that QRJA-based methods offer effective and interpretable ranking predictions.

Abstract

Quantitative Relative Judgment Aggregation (QRJA) is a new research topic in (computational) social choice. In the QRJA model, agents provide judgments on the relative quality of different candidates, and the goal is to aggregate these judgments across all agents. In this work, our main conceptual contribution is to explore the interplay between QRJA in a social choice context and its application to ranking prediction. We observe that in QRJA, judges do not have to be people with subjective opinions; for example, a race can be viewed as a "judgment" on the contestants' relative abilities. This allows us to aggregate results from multiple races to evaluate the contestants' true qualities. At a technical level, we introduce new aggregation rules for QRJA and study their structural and computational properties. We evaluate the proposed methods on data from various real races and show that QRJA-based methods offer effective and interpretable ranking predictions.

Aggregating Quantitative Relative Judgments: From Social Choice to Ranking Prediction

TL;DR

This work introduces new aggregation rules for QRJA and study their structural and computational properties, and evaluates the proposed methods on data from various real races and shows that QRJA-based methods offer effective and interpretable ranking predictions.

Abstract

Quantitative Relative Judgment Aggregation (QRJA) is a new research topic in (computational) social choice. In the QRJA model, agents provide judgments on the relative quality of different candidates, and the goal is to aggregate these judgments across all agents. In this work, our main conceptual contribution is to explore the interplay between QRJA in a social choice context and its application to ranking prediction. We observe that in QRJA, judges do not have to be people with subjective opinions; for example, a race can be viewed as a "judgment" on the contestants' relative abilities. This allows us to aggregate results from multiple races to evaluate the contestants' true qualities. At a technical level, we introduce new aggregation rules for QRJA and study their structural and computational properties. We evaluate the proposed methods on data from various real races and show that QRJA-based methods offer effective and interpretable ranking predictions.
Paper Structure (24 sections, 11 theorems, 24 equations, 11 figures, 1 algorithm)

This paper contains 24 sections, 11 theorems, 24 equations, 11 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivating Examples
  3. Problem Formulation
  4. Theoretical Aspects of $\ell_p$ QRJA
  5. $\ell_p$ QRJA in Almost-Linear Time When $p \geq 1$
  6. NP-Hardness of $\ell_p$ QRJA When $p<1$
  7. Experiments
  8. Experiments Setup
  9. Experiment Results
  10. Related Work
  11. Conclusion
  12. Subsampling Judgments
  13. Subsampling Judgments When $p\in[1,2]$
  14. Subsampling Experiments
  15. Missing Proofs in Section \ref{['sec:computational_aspects']}
  16. ...and 9 more sections

Key Result

Theorem 1

Let $p \ge 1$ be an absolute constant. Consider $\ell_p$ QRJA in def:qrja with loss function $f(t) = t^p$. Assume all input numbers are polynomially bounded in $m$. We can solve $\ell_p$ QRJA in time $O(m^{1+o(1)})$ with $\exp(-\log^c m)$ additive error for any constant $c > 0$.

Figures (11)

  • Figure 1: Bob finishes earlier than Charlie in the Chicago race, which suggests that Bob runs marathons faster than Charlie. However, if we simply calculate the mean or median of all available data, Charlie's mean/median finishing time will be faster than Bob's. This is because, Charlie participated only in the Chicago race, where conditions were more favorable.
  • Figure 2: The same results as in Figure \ref{['fig:example-easy-races']}, but with some data missing. If we only look at the data on Alice and Charlie, it is difficult to judge who is the faster runner. If anything, Charlie appears to be slightly faster. However, if we know Bob's results in these races, then transitivity suggests that Alice runs faster than Charlie.
  • Figure 3: In this example, the races' difficulty has high variance, and everyone's median time is in Boston. Based on this, we would predict Charlie to be faster than Bob. However, if we consider the other two races, overall it seems that Bob runs faster than Charlie.
  • Figure 4: Ordinal accuracy and quantitative loss of the algorithms on all four datasets. Error bars are not shown here as the algorithms are deterministic. The results show that both versions of QRJA perform consistently well across the tested datasets.
  • Figure 5: The performance of $\ell_1$ and $\ell_2$ QRJA on Chess after subsampling judgments using \ref{['alg:subsampling']} with equal weights for all judgments. The subsample rate $\alpha$ means $M = \lfloor \alpha m \rfloor$ in \ref{['alg:subsampling']}. Error bars indicate the standard deviation. The results show that \ref{['alg:subsampling']} can reduce the number of judgments to a factor of $0.4$ with a minor performance loss on Chess.
  • ...and 6 more figures

Theorems & Definitions (14)

  • Definition 1: Quantitative Relative Judgment
  • Definition 2: Quantitative Relative Judgment Aggregation (QRJA)
  • Theorem 1
  • Lemma 1
  • Lemma 2
  • Definition 3: Max-Cut
  • Theorem 2
  • Theorem 3
  • Theorem 3
  • Lemma 3: zhang2019better zhang2019better
  • ...and 4 more