Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy-Regularized Inference: A Predictive Approach

Nicholas G. Polson, Daniel Zantedeschi

TL;DR

This work treats econometric objects as predictive rules, mappings from information to reported predictive distributions, and imposes three structural requirements on evaluation: locality, strict propriety, and coherence under aggregation (coarsening/refinement) of outcome categories.

Abstract

Predictive inference requires balancing statistical accuracy against informational complexity, yet the choice of complexity measure is usually imposed rather than derived. We treat econometric objects as predictive rules, mappings from information to reported predictive distributions, and impose three structural requirements on evaluation: locality, strict propriety, and coherence under aggregation (coarsening/refinement) of outcome categories. These axioms characterize (uniquely, up to affine transformations) the logarithmic score and induce Shannon mutual information (Kullback-Leibler divergence) as the corresponding measure of predictive complexity. The resulting entropy-regularized prediction problem admits Gibbs-form optimal rules, and we establish an essentially complete-class result for the admissible rules we study under joint risk-complexity dominance. Rational inattention emerges as the constrained dual, corresponding to frontier points with binding information capacity. The entropy penalty contributes additive curvature to the predictive criterion; in weakly identified settings, such as weak instruments in IV regression, where the unregularized objective is flat, this curvature stabilizes the predictive criterion. We derive a local quadratic (LAQ) expansion connecting entropy regularization to classical weak-identification diagnostics.

Entropy-Regularized Inference: A Predictive Approach

TL;DR

This work treats econometric objects as predictive rules, mappings from information to reported predictive distributions, and imposes three structural requirements on evaluation: locality, strict propriety, and coherence under aggregation (coarsening/refinement) of outcome categories.

Abstract

Predictive inference requires balancing statistical accuracy against informational complexity, yet the choice of complexity measure is usually imposed rather than derived. We treat econometric objects as predictive rules, mappings from information to reported predictive distributions, and impose three structural requirements on evaluation: locality, strict propriety, and coherence under aggregation (coarsening/refinement) of outcome categories. These axioms characterize (uniquely, up to affine transformations) the logarithmic score and induce Shannon mutual information (Kullback-Leibler divergence) as the corresponding measure of predictive complexity. The resulting entropy-regularized prediction problem admits Gibbs-form optimal rules, and we establish an essentially complete-class result for the admissible rules we study under joint risk-complexity dominance. Rational inattention emerges as the constrained dual, corresponding to frontier points with binding information capacity. The entropy penalty contributes additive curvature to the predictive criterion; in weakly identified settings, such as weak instruments in IV regression, where the unregularized objective is flat, this curvature stabilizes the predictive criterion. We derive a local quadratic (LAQ) expansion connecting entropy regularization to classical weak-identification diagnostics.
Paper Structure (67 sections, 8 theorems, 14 equations, 1 figure, 3 tables)

This paper contains 67 sections, 8 theorems, 14 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Roadmap
  4. Related Literature and Positioning
  5. Empirical likelihood, GEL, and ETEL
  6. Information divergence as regularization
  7. Predictive density aggregation
  8. Running example: Weak instruments.
  9. Why standard Bayesian approaches do not resolve weak IV
  10. Predictive Rules, Scoring, and Information Complexity
  11. Coherence and Uniqueness of the Logarithmic Score
  12. Coarsening, refinement, and additive evaluation
  13. Concrete interpretation of the coherence axioms
  14. Uniqueness of the log score under coherence
  15. Proof sketch (full proof in \ref{['app:logscore']})
  16. ...and 52 more sections

Key Result

Theorem 6

Let $S:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}$ satisfy (i) locality, (ii) strict propriety, and (iii) aggregation coherence. Then there exist constants $c>0$ and a measurable function $d:\mathcal{X}\to\mathbb{R}$ such that

Figures (1)

  • Figure 1: Curvature restoration under entropy regularization. (a) Total curvature $H_\lambda = H_U + \eta H_{I}$ as a function of information price $\eta$ for different first-stage $F$-statistics; dotted lines show unregularized curvature $H_U$. (b) Curvature ratio $H_\lambda / H_U$ showing relative gain from entropy penalty. Under weak instruments ($F \approx 2$), entropy contributes most of the curvature; under strong instruments ($F \approx 50$), the data-driven curvature dominates. $n = 400$, $k = 5$, $\rho = 0.5$, averaged over 50 replications.

Theorems & Definitions (19)

  • Definition 1: Scoring rule
  • Definition 2: Locality
  • Definition 3: Mutual information
  • Definition 4: Entropy-regularized objective
  • Example 5: Failure of aggregation coherence
  • Theorem 6: Uniqueness of the logarithmic score
  • Corollary 7: Mutual information as the coherent refinement gain
  • Theorem 8: Gibbs optimal rule and fixed point
  • Remark 9: Limiting behavior
  • Remark 10: Logit as entropy-regularized prediction
  • ...and 9 more