Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empirical Decision Theory

Christoph Jansen, Georg Schollmeyer, Thomas Augustin, Julian Rodemann

TL;DR

The paper introduces Empirical Decision Problems (EDPs) that obviate the need for explicit state spaces by basing analysis on observed act-consequence protocols. It defines empirical choice functions (ECFs) that operate on subprotocols, and develops three inferential avenues—consistent estimation of population choice sets, robust hypothesis testing, and direct credal inference via contamination models. A key contribution is showing how classical decision rules like expected utility and first-order stochastic dominance can be adapted to this empirical setting, with rigorous guarantees under sampling assumptions and robust extensions for contaminated data. A proof-of-concept evaluates prompting strategies for ChatGPT, illustrating how EDPs can yield inferential statements about AI behavior with robustness analyses. The work lays groundwork for principled, data-driven decision analysis in domains where states are inaccessible or ill-defined, with broad potential applications in AI evaluation and beyond.

Abstract

Analyzing decision problems under uncertainty commonly relies on idealizing assumptions about the describability of the world, with the most prominent examples being the closed world and the small world assumption. Most assumptions are operationalized by introducing states of the world, conditional on which the decision situation can be analyzed without any remaining uncertainty. Conversely, most classical decision-theoretic approaches are not applicable if the states of the world are inaccessible. We propose a decision model that retains the appeal and simplicity of the original theory, but completely overcomes the need to specify the states of the world explicitly. The main idea of our approach is to address decision problems in a radically empirical way: instead of specifying states and consequences prior to the decision analysis, we only assume a protocol of observed act--consequence pairs as model primitives. We show how optimality in such empirical decision problems can be addressed by using protocol-based empirical choice functions and discuss three approaches for deriving inferential guarantees: (I) consistent statistical estimation of choice sets, (II) consistent statistical testing of choice functions with robustness guarantees, and (III) direct inference for empirical choice functions using credal sets. We illustrate our theory with a proof-of-concept application comparing different prompting strategies in generative AI models.

Empirical Decision Theory

TL;DR

The paper introduces Empirical Decision Problems (EDPs) that obviate the need for explicit state spaces by basing analysis on observed act-consequence protocols. It defines empirical choice functions (ECFs) that operate on subprotocols, and develops three inferential avenues—consistent estimation of population choice sets, robust hypothesis testing, and direct credal inference via contamination models. A key contribution is showing how classical decision rules like expected utility and first-order stochastic dominance can be adapted to this empirical setting, with rigorous guarantees under sampling assumptions and robust extensions for contaminated data. A proof-of-concept evaluates prompting strategies for ChatGPT, illustrating how EDPs can yield inferential statements about AI behavior with robustness analyses. The work lays groundwork for principled, data-driven decision analysis in domains where states are inaccessible or ill-defined, with broad potential applications in AI evaluation and beyond.

Abstract

Analyzing decision problems under uncertainty commonly relies on idealizing assumptions about the describability of the world, with the most prominent examples being the closed world and the small world assumption. Most assumptions are operationalized by introducing states of the world, conditional on which the decision situation can be analyzed without any remaining uncertainty. Conversely, most classical decision-theoretic approaches are not applicable if the states of the world are inaccessible. We propose a decision model that retains the appeal and simplicity of the original theory, but completely overcomes the need to specify the states of the world explicitly. The main idea of our approach is to address decision problems in a radically empirical way: instead of specifying states and consequences prior to the decision analysis, we only assume a protocol of observed act--consequence pairs as model primitives. We show how optimality in such empirical decision problems can be addressed by using protocol-based empirical choice functions and discuss three approaches for deriving inferential guarantees: (I) consistent statistical estimation of choice sets, (II) consistent statistical testing of choice functions with robustness guarantees, and (III) direct inference for empirical choice functions using credal sets. We illustrate our theory with a proof-of-concept application comparing different prompting strategies in generative AI models.

Paper Structure

This paper contains 24 sections, 4 theorems, 36 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Motivation and Background
  3. Related Work
  4. Contribution and Outline
  5. The Classical Setting: Decision Problems with Known State Spaces
  6. Decision Making under Uncertainty
  7. Two Extreme (Yet Classical) Choice Functions
  8. Choice Functions Based on Expected Utility Between FSD and EU
  9. Small Worlds, Closed Worlds and the Definition of States
  10. Empirical Decision Problems
  11. The Formal Structure of an EDP
  12. Limitations and Opportunities of Empirical Decision Theory
  13. Statistical Estimation in EDPs
  14. Statistical Testing in EDPs
  15. Testing with i.i.d. Protocols
  16. ...and 9 more sections

Key Result

Theorem 1

Let $ch$ be such that there exists a class $\mathcal{F}$ of uniformly bounded and measurable functions such that $cr$ is uniformly continuous w.r.t. the product pseudometric of the pseudometric $\mathfrak{d}(X,Y):=\sup\nolimits_{f \in \mathcal{F} } | \mathbb{E}( f \circ X - f\circ Y)|$. Note that where $\hat{X}_{a_i}$ and $\hat{X}_{a_j}$ are the random variables defined via the observed outcome

Figures (6)

  • Figure 1: Overview on the concepts discussed in the present paper and how they interconnect with each other. At this point in time, we (essentially) only discussed the very top node of the diagram and will hence now start discussing the paper's core contribution.
  • Figure 2: Choice sets of empirical EU along increasing protocol size. At each stage, the choice sets consists of all colored points that are connected by a vertical gray line.
  • Figure 3: Choice sets of empirical FSD along increasing protocol size. At each stage, the choice sets consists of all colored points that are connected by a vertical gray line.
  • Figure 4: Left: The (approximated) p-values of all six pairwise tests on FSD. Right: The test decisions based on these p-values (where black encodes a reject decision).
  • Figure 5: The p-values of the two pairwise FSD-tests along increasing share of contaminated data points (observed consequences) in the underlying protocol. The dashed red line represents the significance level $\alpha=0.05$.
  • ...and 1 more figures

Theorems & Definitions (18)

  • Example 1
  • Definition 1
  • Example 2
  • Definition 2
  • Definition 3
  • Example 3
  • Example 4
  • Example 5
  • Definition 4
  • Definition 5
  • ...and 8 more