Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical Decision Theory with Counterfactual Loss

Benedikt Koch, Kosuke Imai

TL;DR

This paper broadens statistical decision theory by introducing counterfactual losses that depend on the full set of potential outcomes, enabling evaluation of decisions under all feasible alternatives. It shows that under strong ignorability, the counterfactual risk is identifiable if and only if the counterfactual loss is additive in the potential outcomes, and provides a symbolic linear inverse program to verify identifiability without data. Importantly, additive counterfactual losses can yield different optimal treatment rules from standard losses when the decision space has more than two options, capturing both accuracy and decision difficulty. The framework extends to non-binary treatments and outcomes, offers nonparametric identification results, and equips practitioners with a practical tool to test identifiability of counterfactual risks and decompose them into observable marginals.

Abstract

Many researchers have applied classical statistical decision theory to evaluate treatment choices and learn optimal policies. However, because this framework is based solely on realized outcomes under chosen decisions and ignores counterfactual outcomes, it cannot assess the quality of a decision relative to feasible alternatives. For example, in bail decisions, a judge must consider not only crime prevention but also the avoidance of unnecessary burdens on arrestees. To address this limitation, we generalize standard decision theory by incorporating counterfactual losses, allowing decisions to be evaluated using all potential outcomes. The central challenge in this counterfactual statistical decision framework is identification: since only one potential outcome is observed for each unit, the associated counterfactual risk is generally not identifiable. We prove that, under the assumption of strong ignorability, the counterfactual risk is identifiable if and only if the counterfactual loss function is additive in the potential outcomes. Moreover, we demonstrate that additive counterfactual losses can yield treatment recommendations, which differ from those based on standard loss functions when the decision problem involves more than two treatment options. One interpretation of this result is that additive counterfactual losses can capture the accuracy and difficulty of a decision, whereas standard losses account for accuracy alone. Finally, we formulate a symbolic linear inverse program that, given a counterfactual loss, determines whether its risk is identifiable, without requiring data.

Statistical Decision Theory with Counterfactual Loss

TL;DR

This paper broadens statistical decision theory by introducing counterfactual losses that depend on the full set of potential outcomes, enabling evaluation of decisions under all feasible alternatives. It shows that under strong ignorability, the counterfactual risk is identifiable if and only if the counterfactual loss is additive in the potential outcomes, and provides a symbolic linear inverse program to verify identifiability without data. Importantly, additive counterfactual losses can yield different optimal treatment rules from standard losses when the decision space has more than two options, capturing both accuracy and decision difficulty. The framework extends to non-binary treatments and outcomes, offers nonparametric identification results, and equips practitioners with a practical tool to test identifiability of counterfactual risks and decompose them into observable marginals.

Abstract

Many researchers have applied classical statistical decision theory to evaluate treatment choices and learn optimal policies. However, because this framework is based solely on realized outcomes under chosen decisions and ignores counterfactual outcomes, it cannot assess the quality of a decision relative to feasible alternatives. For example, in bail decisions, a judge must consider not only crime prevention but also the avoidance of unnecessary burdens on arrestees. To address this limitation, we generalize standard decision theory by incorporating counterfactual losses, allowing decisions to be evaluated using all potential outcomes. The central challenge in this counterfactual statistical decision framework is identification: since only one potential outcome is observed for each unit, the associated counterfactual risk is generally not identifiable. We prove that, under the assumption of strong ignorability, the counterfactual risk is identifiable if and only if the counterfactual loss function is additive in the potential outcomes. Moreover, we demonstrate that additive counterfactual losses can yield treatment recommendations, which differ from those based on standard loss functions when the decision problem involves more than two treatment options. One interpretation of this result is that additive counterfactual losses can capture the accuracy and difficulty of a decision, whereas standard losses account for accuracy alone. Finally, we formulate a symbolic linear inverse program that, given a counterfactual loss, determines whether its risk is identifiable, without requiring data.
Paper Structure (43 sections, 16 theorems, 106 equations, 1 figure, 3 tables)

This paper contains 43 sections, 16 theorems, 106 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Literature.
  3. Illustrative Examples
  4. Binary Decisions
  5. Standard Loss
  6. Counterfactual Loss
  7. Additive Counterfactual Loss
  8. Trichotomous Decision
  9. Standard Loss
  10. Counterfactual Loss
  11. Additive Counterfactual Loss
  12. A Numerical Example
  13. Identification Analysis under Strong Ignorability
  14. Setup
  15. Counterfactual Loss and Risk
  16. ...and 28 more sections

Key Result

Theorem 1

Suppose that a counterfactual loss function takes the additive form given in Definition def:additive. Then, under Assumptions ass:iid--ass:ignorability, the conditional counterfactual risk can be decomposed as, where and $\bm{Y}(\mathcal{D})=(Y(0), \ldots, Y(K-1))$. Thus, the difference in the conditional counterfactual risk between any pair of decision rules can be identified.

Figures (1)

  • Figure 1: Numerical example illustrating the decision boundaries under the standard, additive counterfactual, and fully counterfactual loss. The standard loss gives the same decision recommendation as it ignores harm to the public $p$ and harm to the arrestee $r$. In contrast, under the counterfactual loss functions, the decision recommendation varies.

Theorems & Definitions (27)

  • Definition 1: Counterfactual Loss
  • Definition 2: Counterfactual Risk and Conditional Counterfactual Risk
  • Definition 3: Additive Counterfactual Loss
  • Example 1: Counterfactual Classification Loss
  • Example 2: Asymmetric Counterfactual Loss
  • Example 3: Trichotomous decision
  • Definition 4: Identifiability
  • Theorem 1: The Conditional Counterfactual Risk under the Additive Loss
  • Corollary 1: The Counterfactual Risk under the Additive Loss
  • Theorem 2: Additivity as Necessary and Sufficient Condition for Identification
  • ...and 17 more