Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Discrete Filtering with Lookahead or Delay

Pumiao Yan, Jiwon Jeong, Naomi Sagan, Tsachy Weissman

TL;DR

This work extends universal discrete filtering to allow fixed lookahead and delay, showing that any universal SPA combined with an invertible memoryless channel can asymptotically achieve Bayes-optimal filtering. It develops tight excess-loss bounds for causal, delayed, and lookahead settings, linking performance to KL divergence, mutual information, and causally conditioned entropy. The authors instantiate the framework with LZ78-based SPAs, proving their universality and practicality, and implement a complete filtering pipeline with training, Monte Carlo estimation for delay, and lookahead-aware noncausal filtering. Through experiments on a Markov-source scenario, the LZ78-based universal filters approach the Bayesian Bayes limit and outperform a Wiener baseline while offering favorable computational efficiency. The results demonstrate the viability of universal SPAs for real-time, resource-constrained filtering in discrete-sequence environments and lay groundwork for broader noise models and comparisons with neural methods.

Abstract

We consider the universal discrete filtering problem, where an input sequence generated by an unknown source passes through a discrete memoryless channel, and the goal is to estimate its components based on the output sequence with limited lookahead or delay. We propose and establish the universality of a family of schemes for this setting. These schemes are induced by universal Sequential Probability Assignments (SPAs), and inherit their computational properties. We show that the schemes induced by LZ78 are practically implementable and well-suited for scenarios with limited computational resources and latency constraints. In passing, we use some of the intermediate results to obtain upper and lower bounds that appear to be new, in the purely Bayesian setting, on the optimal filtering performance in terms, respectively, of the mutual information between the noise-free and noisy sequence, and the entropy of the noise-free sequence causally conditioned on the noisy one.

Universal Discrete Filtering with Lookahead or Delay

TL;DR

This work extends universal discrete filtering to allow fixed lookahead and delay, showing that any universal SPA combined with an invertible memoryless channel can asymptotically achieve Bayes-optimal filtering. It develops tight excess-loss bounds for causal, delayed, and lookahead settings, linking performance to KL divergence, mutual information, and causally conditioned entropy. The authors instantiate the framework with LZ78-based SPAs, proving their universality and practicality, and implement a complete filtering pipeline with training, Monte Carlo estimation for delay, and lookahead-aware noncausal filtering. Through experiments on a Markov-source scenario, the LZ78-based universal filters approach the Bayesian Bayes limit and outperform a Wiener baseline while offering favorable computational efficiency. The results demonstrate the viability of universal SPAs for real-time, resource-constrained filtering in discrete-sequence environments and lay groundwork for broader noise models and comparisons with neural methods.

Abstract

We consider the universal discrete filtering problem, where an input sequence generated by an unknown source passes through a discrete memoryless channel, and the goal is to estimate its components based on the output sequence with limited lookahead or delay. We propose and establish the universality of a family of schemes for this setting. These schemes are induced by universal Sequential Probability Assignments (SPAs), and inherit their computational properties. We show that the schemes induced by LZ78 are practically implementable and well-suited for scenarios with limited computational resources and latency constraints. In passing, we use some of the intermediate results to obtain upper and lower bounds that appear to be new, in the purely Bayesian setting, on the optimal filtering performance in terms, respectively, of the mutual information between the noise-free and noisy sequence, and the entropy of the noise-free sequence causally conditioned on the noisy one.
Paper Structure (33 sections, 18 theorems, 122 equations, 6 figures, 2 algorithms)

This paper contains 33 sections, 18 theorems, 122 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Notation and Definitions
  4. Properties
  5. Bounding Excess Loss
  6. Causal Estimation
  7. Mismatched Estimation
  8. Estimation with Delay
  9. Estimation with Lookahead
  10. Information Theoretic Bounds on the Bayes Optimal Performance
  11. Universal Filtering Schemes
  12. Finite Uncertainty Set
  13. Parametric Uncertainty Set
  14. Universality with respect to Stationary Sources
  15. Implementation and Practical Considerations
  16. ...and 18 more sections

Key Result

Lemma 2.1

For all $1 \leq t \leq n$, and

Figures (6)

  • Figure 1: A memoryless noisy channel
  • Figure 2: Two-state Markov Source
  • Figure 3: Comparing the MSE loss of universal filter to a linear Wiener filter and the theoretical limit
  • Figure 4: MSE loss of universal filter, increasing the number of Monte Carlo steps
  • Figure 5: Numerical results based on Theorem \ref{['thm:mutual-information-lower-bounds']} where $n=5$
  • ...and 1 more figures

Theorems & Definitions (37)

  • Lemma 2.1: Conditional Probabilities of $X_t$ given the sequence $Z^t$
  • Definition 3.1: Bayes estimator/response
  • Lemma 3.2: Optimal Estimator
  • Definition 3.3: Sequential Probability Assignment, SPA
  • Theorem 3.4: Excess Loss Bound for Causal Estimation
  • Lemma 3.5
  • Theorem 3.6: Excess Loss Bound for Delay
  • Theorem 3.7: Excess Loss Bound for Lookahead
  • Corollary 3.8
  • Theorem 3.9: Mutual information upper bound on the optimal performance
  • ...and 27 more