Variations on the Expectation due to Changes in the Probability Measure

TL;DR

This paper develops closed-form expressions for the variation of the expectation $\mathsf{G}_h$ when the underlying probability measure changes, by leveraging $(h,Q,\lambda)$-Gibbs conditional probability measures. It derives a pointwise characterization $\mathsf{G}_h(x, P_1, P_2)$ and an integrated version $\bar{\mathsf{G}}_h(P_{Y|X}^{(1)}, P_{Y|X}^{(2)}, P_X)$, both expressed through KL divergences to Gibbs measures and marginal divergences, with special cases for Lebesgue and counting reference measures. The results connect to fundamental information measures, revealing that $\bar{\mathsf{G}}_h(P_Y, P_{Y|X}, P_X)$ can be written in terms of mutual information and lautum information, and they unify perspectives across integral probability metrics, distributionally robust optimization, and generalization error analysis. These insights provide a broad, information-theoretic toolkit for analyzing how expectations respond to changes in the underlying probability model across diverse applications.

Abstract

In this paper, closed-form expressions are presented for the variation of the expectation of a given function due to changes in the probability measure used for the expectation. They unveil interesting connections with Gibbs probability measures, mutual information, and lautum information.

Figures (1)

• Figure 1: Geometric interpretation of Lemma \ref{['LemmaDecember20at17h39in2024BusFromSophiaToNice']}, with $Q$ a probability measure.

Theorems & Definitions (11)

• Definition 1
• Definition 2: Gibbs Conditional Probability Measure
• Lemma 1
• Lemma 2
• Lemma 3
• Theorem 4
• Corollary 5
• Theorem 6
• Corollary 7
• Theorem 8