Algebraic Evaluation Theorems
Andrés Corrada-Emmanuel
TL;DR
The paper addresses unsupervised evaluation of experts without ground truth by showing that, with three or more binary jurors, the observed pattern of agreement and disagreement can be encoded into a system of polynomials under an error-independence assumption. It introduces Algebraic Evaluation (AE), which decodes these polynomials to yield two possible juror evaluations and selects the one with greater total accuracy; AE also reports when the independence assumption fails via irrational roots. The work demonstrates this framework both theoretically—through encoding/decoding theorems and algebraic-geometry solutions—and empirically on US Census ACS data, where AE achieves lower labeling error than majority voting and provides empirical uncertainty bounds. It further discusses AI safety implications, such as terminating infinite monitoring chains and addressing super-alignment by enabling evaluation of agents on tasks we may not fully understand. Overall, AE offers a principled, data-sketch-based, non-probabilistic method for evaluating classifiers and graders in unsupervised settings, with a practical alarm for detecting violation of key assumptions.
Abstract
Majority voting (MV) is the prototypical ``wisdom of the crowd'' algorithm. Theorems considering when MV is optimal for group decisions date back to Condorcet's 1785 jury \emph{decision} theorem. The same error independence assumption underlying the theorem can be used to prove a jury \emph{evaluation} theorem that does purely algebraic evaluation (AE) of juror performance based on a batch of their decisions. Three or more binary jurors are enough to obtain the only two possible statistics of their correctness on a test they took. AE is superior to MV in three ways. First, its empirical assumptions are looser and can handle jurors less than 50\% accurate in making decisions. Second, it has point-like precision in evaluating them given its assumption of error independence. This precision enables a multi-accuracy approach that has higher labeling accuracy than MV and comes with empirical uncertainty bounds. And, third, it is self-alarming about the failure of its error independence assumption. Experiments using demographic data from the American Community Survey confirm the practical utility of AE over MV. Two implications of the theorem for AI safety are discussed - a principled way to terminate infinite monitoring chains (who grades the graders?) and the super-alignment problem (how do we evaluate agents doing tasks we do not understand?).