Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exchangeability and randomness for infinite and finite sequences

Vladimir Vovk

TL;DR

This work analyzes the relationship between exchangeability and randomness, showing they are effectively equivalent for infinite sequences via de Finetti representations, but can diverge dramatically for finite sequences, especially with large observation spaces. It reviews De Finetti's theorem, surveys finite-sequence analogues (online vs batch), and presents tight inequalities that bound the discrepancy between exchangeability and IID randomness. The results reveal that, in finite settings, there exist events and predictions that are almost surely possible under exchangeability but become highly significant under randomness, with the worst-case ratio scaling as $N^N/N!$ for infinite alphabets and a computable constant for finite alphabets. The findings have implications for hypothesis testing, conformal-like prediction, and the interpretation of exchangeability in practical finite-sample contexts, while highlighting the conceptual gap between the two notions in finite horizons. Practical impacts include guidance on when de Finetti-type reasoning is appropriate and how tight bounds quantify potential mis-specification in finite-data scenarios.

Abstract

Randomness (in the sense of being generated in an IID fashion) and exchangeability are standard assumptions in nonparametric statistics and machine learning, and relations between them have been a popular topic of research. This short paper draws the reader's attention to the fact that, while for infinite sequences of observations the two assumptions are almost indistinguishable, the difference between them becomes very significant for finite sequences of a given length.

Exchangeability and randomness for infinite and finite sequences

TL;DR

This work analyzes the relationship between exchangeability and randomness, showing they are effectively equivalent for infinite sequences via de Finetti representations, but can diverge dramatically for finite sequences, especially with large observation spaces. It reviews De Finetti's theorem, surveys finite-sequence analogues (online vs batch), and presents tight inequalities that bound the discrepancy between exchangeability and IID randomness. The results reveal that, in finite settings, there exist events and predictions that are almost surely possible under exchangeability but become highly significant under randomness, with the worst-case ratio scaling as for infinite alphabets and a computable constant for finite alphabets. The findings have implications for hypothesis testing, conformal-like prediction, and the interpretation of exchangeability in practical finite-sample contexts, while highlighting the conceptual gap between the two notions in finite horizons. Practical impacts include guidance on when de Finetti-type reasoning is appropriate and how tight bounds quantify potential mis-specification in finite-data scenarios.

Abstract

Randomness (in the sense of being generated in an IID fashion) and exchangeability are standard assumptions in nonparametric statistics and machine learning, and relations between them have been a popular topic of research. This short paper draws the reader's attention to the fact that, while for infinite sequences of observations the two assumptions are almost indistinguishable, the difference between them becomes very significant for finite sequences of a given length.
Paper Structure (11 sections, 3 theorems, 41 equations, 2 figures)

This paper contains 11 sections, 3 theorems, 41 equations, 2 figures.

Table of Contents

  1. Introduction
  2. De Finetti's theorem
  3. Finite sequences of observations
  4. Batch setting
  5. Kolmogorov's and Martin-Löf's work on Bernoulli sequences
  6. Versions of de Finetti's theorem for finite sequences of observations
  7. Tight inequalities
  8. The case of infinite $\mathbf{Z}$
  9. Smaller observation spaces $\mathbf{Z}$
  10. Conclusion
  11. Derivation of $L(1)=1$

Key Result

Proposition 4

For any random variable $F:\mathbf{Z}^N\to[0,\infty)$, Suppose $\left|\mathbf{Z}\right|\ge N$. The bound is tight and attained on a non-zero indicator function. Moreover, there exists an event $A$ in $\mathbf{Z}^N$ such that

Figures (2)

  • Figure 1: The graphs of $\log_{10}C$ for $N=10^3$ (thick red line, with some values given on the left) and $N=10^6$ (thin blue line, with values on the right). The yellow line represents the approximation described in Remark \ref{['rem:limit']}.
  • Figure 2: The graphs of $\log_{10}C$ for $\mathcal{N}=10^3$ (left panel) and $\mathcal{N}=10^6$ (right panel, using $10^5$ as the unit for the labels on both axes), as described in text.

Theorems & Definitions (11)

  • Remark 1
  • Remark 2
  • Remark 3
  • Proposition 4
  • proof
  • Proposition 5
  • proof
  • Lemma 6
  • proof
  • Remark 7
  • ...and 1 more