SP-Rank: A Dataset for Ranked Preferences with Secondary Information

TL;DR

This work introduces SP-Rank, the first large-scale public dataset designed to benchmark ranking algorithms that jointly use first-order votes and second-order meta-predictions. It provides 12,384 datapoints from 1,152 participants across Geography, Movies, and Paintings, encompassing nine elicitation formats and a mix of subset sizes, enabling rigorous evaluation of SP-Voting against traditional vote-only baselines. The authors demonstrate that incorporating second-order information improves ground-truth rank recovery and local subset rankings, and they develop probabilistic models to capture population structure and expert behavior. SP-Rank thus advances preference aggregation, crowd judgment, and human-AI alignment by offering a rich data resource, baseline evaluations, and a path toward learning-to-rank and reward-modeling applications.

Abstract

We introduce $\mathbf{SP-Rank}$, the first large-scale, publicly available dataset for benchmarking algorithms that leverage both first-order preferences and second-order predictions in ranking tasks. Each datapoint includes a personal vote (first-order signal) and a meta-prediction of how others will vote (second-order signal), allowing richer modeling than traditional datasets that capture only individual preferences. SP-Rank contains over 12,000 human-generated datapoints across three domains -- geography, movies, and paintings, and spans nine elicitation formats with varying subset sizes. This structure enables empirical analysis of preference aggregation when expert identities are unknown but presumed to exist, and individual votes represent noisy estimates of a shared ground-truth ranking. We benchmark SP-Rank by comparing traditional aggregation methods that use only first-order votes against SP-Voting, a second-order method that jointly reasons over both signals to infer ground-truth rankings. While SP-Rank also supports models that rely solely on second-order predictions, our benchmarks emphasize the gains from combining both signals. We evaluate performance across three core tasks: (1) full ground-truth rank recovery, (2) subset-level rank recovery, and (3) probabilistic modeling of voter behavior. Results show that incorporating second-order signals substantially improves accuracy over vote-only methods. Beyond social choice, SP-Rank supports downstream applications in learning-to-rank, extracting expert knowledge from noisy crowds, and training reward models in preference-based fine-tuning pipelines. We release the dataset, code, and baseline evaluations (available at https://github.com/amrit19/SP-Rank-Dataset ) to foster research in human preference modeling, aggregation theory, and human-AI alignment.

Figures (3)

• Figure 1: Number of datapoints by domain across each elicitation format. The total number of datapoints is annotated above each format group.
• Figure 2: Accuracy by voting format and domain.
• Figure 3: Distribution of Kendall Tau correlations for rank elicitation format.