Neural Bradley-Terry Rating: Quantifying Properties from Comparisons
Satoru Fujii
TL;DR
NBTR addresses the challenge of quantifying non-metric properties from comparison data by embedding the Bradley-Terry model within a neural network. It supports symmetric and asymmetric (unfair) comparisons through a rating estimator and an advantage adjuster, respectively. Empirical results on a synthetic MNIST-based task and a Pokémon dataset show NBTR can quantify target properties and predict outcomes for unseen items with high correlation and accuracy. The approach enables online, comparison-based quantification suitable for real-world platforms where direct metrics are unavailable.
Abstract
Many properties in the real world don't have metrics and can't be numerically observed, making them difficult to learn. To deal with this challenging problem, prior works have primarily focused on estimating those properties by using graded human scores as the target label in the training. Meanwhile, rating algorithms based on the Bradley-Terry model are extensively studied to evaluate the competitiveness of players based on their match history. In this paper, we introduce the Neural Bradley-Terry Rating (NBTR), a novel machine learning framework designed to quantify and evaluate properties of unknown items. Our method seamlessly integrates the Bradley-Terry model into the neural network structure. Moreover, we generalize this architecture further to asymmetric environments with unfairness, a condition more commonly encountered in real-world settings. Through experimental analysis, we demonstrate that NBTR successfully learns to quantify and estimate desired properties.