HITSnDIFFs: From Truth Discovery to Ability Discovery by Recovering Matrices with the Consecutive Ones Property
Zixuan Chen, Subhodeep Mitra, R Ravi, Wolfgang Gatterbauer
TL;DR
Ability discovery ranks users by their latent ability to choose correct labels across heterogeneous items, framed as a dual of truth discovery. The authors introduce HITSnDIFFs (HnD), a spectral method that leverages the Consecutive Ones Property ($C1P$) and Item Response Theory (IRT) to recover the correct user ordering in ideal cases and to robustly handle non-ideal data. They show that HnD reconstructs the $C1P$ ordering via the second-largest eigenvector of a row-normalized update matrix and outperform ABH in scalability and stability, while providing entropy-based symmetry breaking to resolve order direction. Extensive synthetic-IRT experiments and real-world MCQ datasets demonstrate superior accuracy and linear scalability in the number of users and questions, with practical implications for crowd-sourced ability assessment and ranking.
Abstract
We analyze a general problem in a crowd-sourced setting where one user asks a question (also called item) and other users return answers (also called labels) for this question. Different from existing crowd sourcing work which focuses on finding the most appropriate label for the question (the "truth"), our problem is to determine a ranking of the users based on their ability to answer questions. We call this problem "ability discovery" to emphasize the connection to and duality with the more well-studied problem of "truth discovery". To model items and their labels in a principled way, we draw upon Item Response Theory (IRT) which is the widely accepted theory behind standardized tests such as SAT and GRE. We start from an idealized setting where the relative performance of users is consistent across items and better users choose better fitting labels for each item. We posit that a principled algorithmic solution to our more general problem should solve this ideal setting correctly and observe that the response matrices in this setting obey the Consecutive Ones Property (C1P). While C1P is well understood algorithmically with various discrete algorithms, we devise a novel variant of the HITS algorithm which we call "HITSNDIFFS" (or HND), and prove that it can recover the ideal C1P-permutation in case it exists. Unlike fast combinatorial algorithms for finding the consecutive ones permutation (if it exists), HND also returns an ordering when such a permutation does not exist. Thus it provides a principled heuristic for our problem that is guaranteed to return the correct answer in the ideal setting. Our experiments show that HND produces user rankings with robustly high accuracy compared to state-of-the-art truth discovery methods. We also show that our novel variant of HITS scales better in the number of users than ABH, the only prior spectral C1P reconstruction algorithm.