Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Letters to Words and Back: Invertible Coding of Stationary Measures

Łukasz Dębowski

TL;DR

This work constructs an invertible, stationarity- and ergodicity-preserving mapping between probability measures on word- and letter-like infinite sequences via self-avoiding codes, termed the normalized transport. The authors show global and trajectory-based formulations that recover and invert each other, connect to asymptotically mean stationary measures, and preserve ergodic decompositions. They prove ergodicity of successive recurrence times under ergodic measures and derive a precise relation between entropy rates across the transport, tying the entropy to the average code-length via $h_Y= h_X / \int L_g dP_X$. The framework offers a principled bridge between textual representations and provides tools for analyzing information rates and recurrence phenomena in stationary processes, with potential applications in statistical language modeling and formal linguistics.

Abstract

Motivated by problems of statistical language modeling, we consider probability measures on infinite sequences over two countable alphabets of a different cardinality, such as letters and words. We introduce an invertible mapping between such measures, called the normalized transport, that preserves both stationarity and ergodicity. The normalized transport applies so called self-avoiding codes that generalize comma-separated codes and specialize bijective stationary codes. The normalized transport is also connected to the usual measure transport via underlying asymptotically mean stationary measures. It preserves the ergodic decomposition. The normalized transport and self-avoiding codes arise in the problem of successive recurrence times. We show that successive recurrence times are ergodic for an ergodic measure, which strengthens a result by Chen Moy from 1959. We also relate the entropy rates of processes linked by the normalized transport.

From Letters to Words and Back: Invertible Coding of Stationary Measures

TL;DR

This work constructs an invertible, stationarity- and ergodicity-preserving mapping between probability measures on word- and letter-like infinite sequences via self-avoiding codes, termed the normalized transport. The authors show global and trajectory-based formulations that recover and invert each other, connect to asymptotically mean stationary measures, and preserve ergodic decompositions. They prove ergodicity of successive recurrence times under ergodic measures and derive a precise relation between entropy rates across the transport, tying the entropy to the average code-length via . The framework offers a principled bridge between textual representations and provides tools for analyzing information rates and recurrence phenomena in stationary processes, with potential applications in statistical language modeling and formal linguistics.

Abstract

Motivated by problems of statistical language modeling, we consider probability measures on infinite sequences over two countable alphabets of a different cardinality, such as letters and words. We introduce an invertible mapping between such measures, called the normalized transport, that preserves both stationarity and ergodicity. The normalized transport applies so called self-avoiding codes that generalize comma-separated codes and specialize bijective stationary codes. The normalized transport is also connected to the usual measure transport via underlying asymptotically mean stationary measures. It preserves the ergodic decomposition. The normalized transport and self-avoiding codes arise in the problem of successive recurrence times. We show that successive recurrence times are ergodic for an ergodic measure, which strengthens a result by Chen Moy from 1959. We also relate the entropy rates of processes linked by the normalized transport.
Paper Structure (16 sections, 13 theorems, 52 equations, 3 figures)

This paper contains 16 sections, 13 theorems, 52 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Self-avoiding codes
  4. Normalized transport
  5. Proof of Theorem \ref{['theoEnsemble']}
  6. Mutual invertibility
  7. Conservation of stationarity
  8. Conservation of ergodicity
  9. Equality (\ref{['Normalization']})
  10. Proof of Theorem \ref{['theoTrajectoryAMS']}
  11. Mutual invertibility
  12. Conservation of the AMS property
  13. Stationary means
  14. Recurrence times
  15. Entropy rate
  16. ...and 1 more sections

Key Result

Proposition 1

A measurable function $g:\Omega_X\to\Omega_Y$ is a code adjusted to a function $L:\Omega_X\to\mathbb{N}$ if and only if $g$ is a self-avoiding code and its quasi-period equals $L_g=L$.

Figures (3)

  • Figure 1: Finitary codes $\mathbb{X} \to \mathbb{Y}^*$.
  • Figure 2: Infinitary codes $\Omega_X \to \Omega_Y$.
  • Figure 3: Infinitary codes $\Omega_X \to \Omega_W$, where $\mathbb{W} = \mathbb{Y}^*$.

Theorems & Definitions (33)

  • Example 1: English "not"
  • Definition 1: quasi-period, cf. Debowski21
  • Definition 2: quasi-periodic code, Debowski21
  • Definition 3: self-avoiding code
  • Example 2: code extension, Debowski21
  • Example 3: comma-separated code, Debowski21
  • Example 4: comma-embedded code
  • Example 5: Santa Fe and Oracle processes, Debowski11bDebowski21b
  • Example 6: shifted code
  • Definition 4: accumulations
  • ...and 23 more