Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Refereed Learning

Ran Canetti, Ephraim Linder, Connor Wagaman

TL;DR

This work introduces refereed learning, a framework in which a learner-verifier uses two competing provers, one honest, to assess properties of opaque black-box models with minimal ground-truth access. The authors develop a suite of tools, notably certifiable sampling, certifiable sum, and certifiable index, to enable high-precision model selection and loss estimation under general distributions and loss functions. They present protocols achieving near-optimal loss bounds with a single ground-truth query and poly(d) communication, along with lower bounds that prove necessary access to the ground-truth distribution and exponential prover-time in certain settings. The results extend to general metric losses and arbitrary distributions, with efficient protocols for juntas and precise handling of precision, making refereed learning a robust framework for verifiable evaluation of complex models. Overall, the paper provides a principled, scalable approach to validating claims about opaque models using competitive, partly trusted agents, with clear trade-offs between prover power, sample complexity, and communication.

Abstract

We initiate an investigation of learning tasks in a setting where the learner is given access to two competing provers, only one of which is honest. Specifically, we consider the power of such learners in assessing purported properties of opaque models. Following prior work that considers the power of competing provers in different settings, we call this setting refereed learning. After formulating a general definition of refereed learning tasks, we show refereed learning protocols that obtain a level of accuracy that far exceeds what is obtainable at comparable cost without provers, or even with a single prover. We concentrate on the task of choosing the better one out of two black-box models, with respect to some ground truth. While we consider a range of parameters, perhaps our most notable result is in the high-precision range: For all $\varepsilon>0$ and ambient dimension $d$, our learner makes only one query to the ground truth function, communicates only $(1+\frac{1}{\varepsilon^2})\cdot\text{poly}(d)$ bits with the provers, and outputs a model whose loss is within a multiplicative factor of $(1+\varepsilon)$ of the best model's loss. Obtaining comparable loss with a single prover would require the learner to access the ground truth at almost all of the points in the domain. To obtain this bound, we develop a technique that allows the learner to sample, using the provers, from a distribution that is not efficiently samplable to begin with. We find this technique to be of independent interest. We also present lower bounds that demonstrate the optimality of our protocols in a number of respects, including prover complexity, number of samples, and need for query access.

Refereed Learning

TL;DR

This work introduces refereed learning, a framework in which a learner-verifier uses two competing provers, one honest, to assess properties of opaque black-box models with minimal ground-truth access. The authors develop a suite of tools, notably certifiable sampling, certifiable sum, and certifiable index, to enable high-precision model selection and loss estimation under general distributions and loss functions. They present protocols achieving near-optimal loss bounds with a single ground-truth query and poly(d) communication, along with lower bounds that prove necessary access to the ground-truth distribution and exponential prover-time in certain settings. The results extend to general metric losses and arbitrary distributions, with efficient protocols for juntas and precise handling of precision, making refereed learning a robust framework for verifiable evaluation of complex models. Overall, the paper provides a principled, scalable approach to validating claims about opaque models using competitive, partly trusted agents, with clear trade-offs between prover power, sample complexity, and communication.

Abstract

We initiate an investigation of learning tasks in a setting where the learner is given access to two competing provers, only one of which is honest. Specifically, we consider the power of such learners in assessing purported properties of opaque models. Following prior work that considers the power of competing provers in different settings, we call this setting refereed learning. After formulating a general definition of refereed learning tasks, we show refereed learning protocols that obtain a level of accuracy that far exceeds what is obtainable at comparable cost without provers, or even with a single prover. We concentrate on the task of choosing the better one out of two black-box models, with respect to some ground truth. While we consider a range of parameters, perhaps our most notable result is in the high-precision range: For all and ambient dimension , our learner makes only one query to the ground truth function, communicates only bits with the provers, and outputs a model whose loss is within a multiplicative factor of of the best model's loss. Obtaining comparable loss with a single prover would require the learner to access the ground truth at almost all of the points in the domain. To obtain this bound, we develop a technique that allows the learner to sample, using the provers, from a distribution that is not efficiently samplable to begin with. We find this technique to be of independent interest. We also present lower bounds that demonstrate the optimality of our protocols in a number of respects, including prover complexity, number of samples, and need for query access.

Paper Structure

This paper contains 42 sections, 12 theorems, 26 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. This work
  3. Defining refereed learning
  4. Bounding the expected score and loss.
  5. On strategic provers.
  6. Our results
  7. Protocols.
  8. Lower bounds.
  9. Our techniques
  10. Protocols for purely additive and for mixed errors.
  11. Protocols for purely multiplicative error.
  12. Certifiable uniform sampling.
  13. The certifiable sum protocol.
  14. The certifiable index protocol.
  15. Extending to general sample distributions.
  16. ...and 27 more sections

Key Result

Lemma 3.1

There exists a protocol ${\left [ {{\mathcal{P}}_0,{\mathcal{P}}_1,\mathcal{V}} \right]}$ such that for all $d\in\mathbb N$, distributions ${\mathcal{D}}$ over $\{0,1\}^d$ with probability mass function $Q_{\mathcal{D}}$, distance $\delta\in (0,1)$, and sample size $m\in\mathbb N$, there exists a di

Theorems & Definitions (46)

  • Definition 1.1: Refereed learning, general case (informal)
  • Definition 1.2: Refereed learning, loss minimization (informal)
  • Definition 2.1: Refereed learning protocol---general case
  • Definition 2.2: Metric loss function, zero-one metric
  • Definition 2.3: Refereed learning protocol---loss minimization
  • Definition 2.4: Set $\mathbb Q_{\lambda}$, $\lambda$-precise, distribution family $\mathds D_{\lambda}$
  • Definition 2.5: Families $\mathfrak F$ and $\mathfrak D$
  • Lemma 3.1: Certifiable sample
  • Lemma 3.2: Certifiable sum
  • proof : Proof of \ref{['lem:cert_sample']}
  • ...and 36 more