Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Value of Context: Human versus Black Box Evaluators

Andrei Iakovlev, Annie Liang

TL;DR

A comparison of evaluation by algorithms with more traditional evaluation by human experts to help individuals compare evaluation by algorithms with more traditional evaluation by human experts.

Abstract

Machine learning algorithms are now capable of performing evaluations previously conducted by human experts (e.g., medical diagnoses). How should we conceptualize the difference between evaluation by humans and by algorithms, and when should an individual prefer one over the other? We propose a framework to examine one key distinction between the two forms of evaluation: Machine learning algorithms are standardized, fixing a common set of covariates by which to assess all individuals, while human evaluators customize which covariates are acquired to each individual. Our framework defines and analyzes the advantage of this customization -- the value of context -- in environments with high-dimensional data. We show that unless the agent has precise knowledge about the joint distribution of covariates, the benefit of additional covariates generally outweighs the value of context.

The Value of Context: Human versus Black Box Evaluators

TL;DR

A comparison of evaluation by algorithms with more traditional evaluation by human experts to help individuals compare evaluation by algorithms with more traditional evaluation by human experts.

Abstract

Machine learning algorithms are now capable of performing evaluations previously conducted by human experts (e.g., medical diagnoses). How should we conceptualize the difference between evaluation by humans and by algorithms, and when should an individual prefer one over the other? We propose a framework to examine one key distinction between the two forms of evaluation: Machine learning algorithms are standardized, fixing a common set of covariates by which to assess all individuals, while human evaluators customize which covariates are acquired to each individual. Our framework defines and analyzes the advantage of this customization -- the value of context -- in environments with high-dimensional data. We show that unless the agent has precise knowledge about the joint distribution of covariates, the benefit of additional covariates generally outweighs the value of context.
Paper Structure (43 sections, 16 theorems, 149 equations, 1 figure, 1 table)

This paper contains 43 sections, 16 theorems, 149 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Literature
  3. (Asymptotic) Learning
  4. Strategic Information Disclosure
  5. Human vs AI Evaluation
  6. Model
  7. Setup
  8. Evaluation of the agent
  9. Value of context
  10. Model Uncertainty
  11. Expected Value of Context
  12. Main Results
  13. Example
  14. The Expected Value of Context
  15. Human versus Black Box
  16. ...and 28 more sections

Key Result

Theorem 3.1

Suppose Assumptions assp:Exchangeability and assp:ConstantVarY hold. Then for every covariate vector $\bold{x} \in \{0,1\}^\infty$, the expected value of context vanishes to zero as $n$ grows large, i.e.,

Figures (1)

  • Figure 1: Let $C=100$ and $\alpha_h=0.1$. Then the comparisons in Theorem \ref{['prop:PreferHA']} hold for all $n \geq N$ as depicted here.

Theorems & Definitions (48)

  • Example 1: Job Interview
  • Example 2: Medical Prediction
  • Example 3: Higher Evaluations are Better
  • Example 4: More Accurate Evaluations are Better
  • Definition 2.1: Value of Context
  • Example 5: High Value of Context
  • Example 6: Low Value of Context
  • Example 7
  • Example 8
  • Example 9: Only One Covariate is Relevant
  • ...and 38 more