Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using tournaments to calculate AUROC for zero-shot classification with LLMs

WonJin Yoon, Ian Bulovic, Timothy A. Miller

TL;DR

The paper tackles the challenge of fairly evaluating zero-shot LLM classifiers by reframing binary tasks as pairwise comparisons and using Elo ratings to rank dataset instances, enabling threshold-free AUROC estimation. It introduces a tournament-based method that prompts LLMs to compare pairs, updates instance Elo scores, and derives a global ranking for ROC/PR analysis, along with baseline zero-shot prompts for comparison. Scheduling strategies (Random, Graph, Swiss/Dutch) are examined to minimize the number of comparisons while maintaining ranking quality. Experiments on CoLA, CliniFact, and LCD bench show improved AUROC and F1 over baselines, indicating practical benefits for threshold-independent evaluation of zero-shot LLMs.

Abstract

Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.

Using tournaments to calculate AUROC for zero-shot classification with LLMs

TL;DR

The paper tackles the challenge of fairly evaluating zero-shot LLM classifiers by reframing binary tasks as pairwise comparisons and using Elo ratings to rank dataset instances, enabling threshold-free AUROC estimation. It introduces a tournament-based method that prompts LLMs to compare pairs, updates instance Elo scores, and derives a global ranking for ROC/PR analysis, along with baseline zero-shot prompts for comparison. Scheduling strategies (Random, Graph, Swiss/Dutch) are examined to minimize the number of comparisons while maintaining ranking quality. Experiments on CoLA, CliniFact, and LCD bench show improved AUROC and F1 over baselines, indicating practical benefits for threshold-independent evaluation of zero-shot LLMs.

Abstract

Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.

Paper Structure

This paper contains 22 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Methods
  4. Baseline
  5. Tournament-based method
  6. Tournament setup
  7. Scheduling strategies
  8. Evaluation
  9. Metrics
  10. AUROC
  11. Datasets
  12. CoLA
  13. CliniFact
  14. LCD Benchmark (Summary)
  15. Results
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Figures showing AUROC improvements across rounds for each dataset and model.
  • Figure 2: Figures showing LCD bench (Summ) results on Llama3.2-Instruct. Cloud-hosted models were not evaluated on this dataset in compliance with data usage restrictions.
  • Figure 3: Figures showing AUROC improvements across rounds for CoLA.
  • Figure 4: Figures showing AUROC improvements across rounds for CliniFact.
  • Figure 5: Figures showing AUROC improvements across rounds for LCD bench (Summ).