When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery

TL;DR

The paper challenges the assumption that ordinal paired comparisons always yield stronger ranking than binary ones, showing that binarization can lead to faster, exponential convergence in ranking recovery under a counting algorithm. It introduces a flexible generalized additive ordinal model with a strength link function $\phi$ and a pattern function $\psi_\gamma$, clarifying when binarization helps via a signal-to-noise analysis of the ordinal pattern. Theoretical results show a quantifiable gap in ranking accuracy between binary and ordinal data, controlled by the SNR of the ordinal pattern, and identify pattern forms that minimize SNR to maximize binarization gains. These insights are validated through extensive simulations and a MovieLens real-data application, with practical implications for designing feedback mechanisms and ranking algorithms that leverage binary responses for more robust preference recovery.

Abstract

Paired comparison data, where users evaluate items in pairs, play a central role in ranking and preference learning tasks. While ordinal comparison data intuitively offer richer information than binary comparisons, this paper challenges that conventional wisdom. We propose a general parametric framework for modeling ordinal paired comparisons without ties. The model adopts a generalized additive structure, featuring a link function that quantifies the preference difference between two items and a pattern function that governs the distribution over ordinal response levels. This framework encompasses classical binary comparison models as special cases, by treating binary responses as binarized versions of ordinal data. Within this framework, we show that binarizing ordinal data can significantly improve the accuracy of ranking recovery. Specifically, we prove that under the counting algorithm, the ranking error associated with binary comparisons exhibits a faster exponential convergence rate than that of ordinal data. Furthermore, we characterize a substantial performance gap between binary and ordinal data in terms of a signal-to-noise ratio (SNR) determined by the pattern function. We identify the pattern function that minimizes the SNR and maximizes the benefit of binarization. Extensive simulations and a real application on the MovieLens dataset further corroborate our theoretical findings.

Figures (11)

• Figure 1: Two users submit pairwise comparison responses using two distinct sets of system-defined options. In Scenario I (Left), a preference ranking tie occurs. In Scenario II (Right), object 1 receives more favorable feedback.
• Figure 2: Four examples for $\phi(x)$: (1) $\phi(x)=x^3$; (2) $\phi(x)=x$; (3) $\phi(x)=\frac{1-e^{-x}}{1+e^{-x}}$; (4) $\phi(x)=\mathop{\mathrm{sign}}(x)|x|^{1/3}$.
• Figure 3: The distributions of ordinal comparison data derived from three real datasets.
• Figure 4: A comparison between $\mathbb{P}(A > 0)$ and $\mathbb{P}(B > 0)$ is conducted under the proposed model, where the propensity functions are specified as $\phi(x) = x$ and $\psi_{\gamma}(k) = -\beta |k|$. When $\beta = 0.1$, the SNR of $X_{\gamma_{12}^\star}$ is 4.5523, whereas for $\beta = 0.9$, the SNR of $X_{\gamma_{12}^\star}$ decreases to 3.5723.
• Figure 5: A comparison between $\mathbb{E}[\tau(\bm{S},\bm{\theta}^\star)]$ and $\mathbb{E}[\tau(\widetilde{\bm{S}},\bm{\theta}^\star)]$ under the proposed model, with propensity functions with $\psi_{\gamma}(x) = -0.1|x|+0.5\sqrt{|k\gamma|}$ (top) and $\psi_{\gamma}(x) = -0.9|x|+0.5\sqrt{|k\gamma|}$ (bottom).