Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation
Pius von Däniken, Jan Deriu, Don Tuggener, Mark Cieliebak
TL;DR
The paper addresses the problem that trained metrics for evaluating generative AI outputs can systematically favor certain systems, distorting rankings. It defines favoritism formally and introduces the Favi-Score, a simple, interpretable measure computed from the confusion matrix of human vs metric preference ratings and an error-cost matrix, with a formal equivalence to the change in outcome margins per error, ensuring a range of [-2,2]. The authors apply Favi-Score to data from chatbot, summarization, and machine translation tasks, showing that all metrics exhibit some favoritism and that Favi-Score provides diagnostic insight beyond traditional sign-accuracy, aiding more reliable interpretation of system rankings. They advocate using Favi-Score alongside sign-accuracy to better understand and mitigate biases in automated evaluation of generative AI, with implications for fairer and more robust model comparisons.
Abstract
Generative AI systems have become ubiquitous for all kinds of modalities, which makes the issue of the evaluation of such models more pressing. One popular approach is preference ratings, where the generated outputs of different systems are shown to evaluators who choose their preferences. In recent years the field shifted towards the development of automated (trained) metrics to assess generated outputs, which can be used to create preference ratings automatically. In this work, we investigate the evaluation of the metrics themselves, which currently rely on measuring the correlation to human judgments or computing sign accuracy scores. These measures only assess how well the metric agrees with the human ratings. However, our research shows that this does not tell the whole story. Most metrics exhibit a disagreement with human system assessments which is often skewed in favor of particular text generation systems, exposing a degree of favoritism in automated metrics. This paper introduces a formal definition of favoritism in preference metrics, and derives the Favi-Score, which measures this phenomenon. In particular we show that favoritism is strongly related to errors in final system rankings. Thus, we propose that preference-based metrics ought to be evaluated on both sign accuracy scores and favoritism.