PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation

TL;DR

PRECISE addresses the annotation bottleneck in evaluating ranking systems by combining minimal human labels with LLM judgments to debias metric estimates. It extends Prediction-Powered Inference to sub-instance ranking tasks by reformulating the top-K relevance problem into a tractable K-hot vector framework and using a calibrated annotator to produce unbiased, low-variance estimates for metrics like Precision@K. The approach achieves substantial variance reduction with small gold sets, identifies optimal unlabeled data sizes around 100× for cost efficiency, and demonstrates strong alignment between offline PRECISE estimates and online production improvements in a real-world e-commerce setting. This enables scalable, bias-corrected evaluation of ML-driven ranking and query reformulation systems with practical deployment benefits and broad applicability to dynamic and multi-modal retrieval scenarios.

Abstract

Evaluating the quality of search, ranking and RAG systems traditionally requires a significant number of human relevance annotations. In recent times, several deployed systems have explored the usage of Large Language Models (LLMs) as automated judges for this task while their inherent biases prevent direct use for metric estimation. We present a statistical framework extending Prediction-Powered Inference (PPI) that combines minimal human annotations with LLM judgments to produce reliable estimates of metrics which require sub-instance annotations. Our method requires as few as 100 human-annotated queries and 10,000 unlabeled examples, reducing annotation requirements significantly compared to traditional approaches. We formulate our proposed framework (PRECISE) for inference of relevance uplift for an LLM-based query reformulation application, extending PPI to sub-instance annotations at the query-document level. By reformulating the metric-integration space, we reduced the computational complexity from O(2^|C|) to O(2^K), where |C| represents corpus size (in order of millions). Detailed experiments across prominent retrieval datasets demonstrate that our method reduces the variance of estimates for the business-critical Precision@K metric, while effectively correcting for LLM bias in low-resource settings.

Figures (4)

• Figure 1: Code-mixed queries encountered in our production system, demonstrating the linguistic challenges of Indian e-commerce search. Left: queries from customers often mix Hindi words written in Latin script with English. Right: query-reformulation into grammatical English using a frontier LLM greatly improves search relevance. Our deployed approach PRECISE-PPI seeks to estimate the performance of the query-reformulation approach by debiasing LLM relevance judgements with minimal human annotations.
• Figure 2: High-level flow of our PRECISE-PPI method to estimate relevance metrics. Our approach combines estimates from LLM annotations on unlabelled queries and human-labelled gold annotations of query-product relevance.
• Figure 3: Estimated Precision@4 on ESCI. We show sampling distributions and 95% CI for different estimators, calculated by sampling 50 gold datasets from ESCI. We consider samples of size $n=30$ (top row) and $n=100$, using $N=60,000$ unlabeled queries. Claude 3 Sonnet is used as the calibrated annotation model. The vertical yellow line denotes the true relevance, averaged across the entire ESCI dataset. PRECISE-PPI estimator (green) achieves variance reduction compared to the estimator using only Gold data (red), with superior reduction at higher $\lambda$ values. Both these approaches are significantly less biased than the LLM-only annotators prob (cyan) and bin (cerulean).
• Figure 4: Calibration comparison across LLM evaluator models. Left: Claude 3 Sonnet (well-calibrated), Center: Claude 3 Haiku (moderately calibrated), Right: Jina Turbo (poorly calibrated). Blue bars represent true positives, red bars represent true negatives.