Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bias Detection Via Signaling

Yiling Chen, Tao Lin, Ariel D. Procaccia, Aaditya Ramdas, Itai Shapira

TL;DR

This work measures an agent's bias by designing a signaling scheme and observing the actions they take in response to different signals, assuming that they are maximizing their own expected utility; the goal is to detect bias with a minimum number of signals.

Abstract

We introduce and study the problem of detecting whether an agent is updating their prior beliefs given new evidence in an optimal way that is Bayesian, or whether they are biased towards their own prior. In our model, biased agents form posterior beliefs that are a convex combination of their prior and the Bayesian posterior, where the more biased an agent is, the closer their posterior is to the prior. Since we often cannot observe the agent's beliefs directly, we take an approach inspired by information design. Specifically, we measure an agent's bias by designing a signaling scheme and observing the actions they take in response to different signals, assuming that they are maximizing their own expected utility; our goal is to detect bias with a minimum number of signals. Our main results include a characterization of scenarios where a single signal suffices and a computationally efficient algorithm to compute optimal signaling schemes.

Bias Detection Via Signaling

TL;DR

This work measures an agent's bias by designing a signaling scheme and observing the actions they take in response to different signals, assuming that they are maximizing their own expected utility; the goal is to detect bias with a minimum number of signals.

Abstract

We introduce and study the problem of detecting whether an agent is updating their prior beliefs given new evidence in an optimal way that is Bayesian, or whether they are biased towards their own prior. In our model, biased agents form posterior beliefs that are a convex combination of their prior and the Bayesian posterior, where the more biased an agent is, the closer their posterior is to the prior. Since we often cannot observe the agent's beliefs directly, we take an approach inspired by information design. Specifically, we measure an agent's bias by designing a signaling scheme and observing the actions they take in response to different signals, assuming that they are maximizing their own expected utility; our goal is to detect bias with a minimum number of signals. Our main results include a characterization of scenarios where a single signal suffices and a computationally efficient algorithm to compute optimal signaling schemes.
Paper Structure (21 sections, 11 theorems, 46 equations, 1 figure, 1 algorithm)

This paper contains 21 sections, 11 theorems, 46 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model
  3. Warm-Up: A Two-State, Two-Action Example
  4. Computing the Optimal Signaling Scheme in the General Case
  5. Optimality of Constant Signaling Schemes
  6. Revelation Principle
  7. Algorithm for Computing the Optimal Signaling Scheme
  8. Geometric Characterization of the Testability of Bias
  9. Discussion
  10. Missing Proofs from Section \ref{['sec:optimal_signaling_scheme']}
  11. Proof of Lemma \ref{['lem:adaptive-external-to-boundary']}
  12. Proof of Lemma \ref{['lem:internal-not-useful']}
  13. Proof of Lemma \ref{['lem:revelation-principle']}
  14. Missing Proofs from Section \ref{['sec:geo_char']}
  15. Proof of Lemma \ref{['lem:I_a_tau_translation']}
  16. ...and 6 more sections

Key Result

Lemma 2.1

Let $\pi$ be a signaling scheme where each signal $s \in S$ is sent with unconditional probability $\pi(s) = \sum_{\theta \in \Theta} \mu_0(\theta) \pi(s|\theta)$ and induces true posterior $\mu_s$. Then, the prior $\mu_0$ equals the convex combination of $\{ \mu_s \}_{s \in S}$ with weights $\{\pi(

Figures (1)

  • Figure 1: The three qualitatively different cases for detecting the level of bias, each illustrated within a simplex over three states where $\mu_0$ is the prior belief. Each point in the simplex corresponds to an optimal action for the agent. Green curves indicate indifference between the default action $a_0$ and another action under an unbiased belief. Orange curves are translated versions of these indifference curves; a posterior on these curves means the agent's biased belief (at bias level $\tau$) aligns with the green curves. From (a) to (c), $\tau$ increases, translating the orange curves further. In \ref{['fig:1']}, $\mu_0$ can be represented as a convex combination of points on the translated curves, allowing bias level detection with a single sample. In \ref{['fig:2']}, only some signals are useful, requiring more than one sample in the worst case. In \ref{['fig:3']}, the bias level cannot be tested against $\tau$.

Theorems & Definitions (25)

  • Definition 2.1: sample complexity
  • Lemma 2.1: Splitting Lemma, e.g., kamenica_bayesian_2011
  • Theorem 3.1
  • proof
  • Lemma 4.1
  • Definition 4.1
  • Lemma 4.2
  • proof
  • Lemma 4.3
  • Lemma 4.4
  • ...and 15 more