Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Expected Kullback-Leibler-based characterizations of score-driven updates

Ramon de Punder, Timo Dimitriadis, Rutger-Jan Lange

TL;DR

This work provides an information-theoretic characterization of SD updates based on reductions in the expected Kullback-Leibler (EKL) divergence relative to the true -- but unknown -- data-generating density, and establishes EKL as their natural information-theoretic foundation.

Abstract

Score-driven (SD) models are a standard tool in statistics and econometrics, with applications in hundreds of published articles in the past decade. We provide an information-theoretic characterization of SD updates based on reductions in the expected Kullback-Leibler (EKL) divergence relative to the true -- but unknown -- data-generating density. EKL reductions occur if and only if the expected update direction aligns with the expected score; i.e., their inner product should be positive. This equivalence condition uniquely identifies SD updates (including scaled or clipped variants) as being EKL reducing, even in non-concave, multivariate, and misspecified settings. We further derive explicit bounds on admissible learning rates in terms of score moments, linking SD methods to adaptive optimization techniques. By contrast, alternative performance measures in the literature impose stronger conditions (e.g., concave logarithmic densities) and do not characterize SD updates: other updating rules may improve these measures, while SD updates need not. Our results provide a rigorous justification for SD models and establish EKL as their natural information-theoretic foundation.

Expected Kullback-Leibler-based characterizations of score-driven updates

TL;DR

This work provides an information-theoretic characterization of SD updates based on reductions in the expected Kullback-Leibler (EKL) divergence relative to the true -- but unknown -- data-generating density, and establishes EKL as their natural information-theoretic foundation.

Abstract

Score-driven (SD) models are a standard tool in statistics and econometrics, with applications in hundreds of published articles in the past decade. We provide an information-theoretic characterization of SD updates based on reductions in the expected Kullback-Leibler (EKL) divergence relative to the true -- but unknown -- data-generating density. EKL reductions occur if and only if the expected update direction aligns with the expected score; i.e., their inner product should be positive. This equivalence condition uniquely identifies SD updates (including scaled or clipped variants) as being EKL reducing, even in non-concave, multivariate, and misspecified settings. We further derive explicit bounds on admissible learning rates in terms of score moments, linking SD methods to adaptive optimization techniques. By contrast, alternative performance measures in the literature impose stronger conditions (e.g., concave logarithmic densities) and do not characterize SD updates: other updating rules may improve these measures, while SD updates need not. Our results provide a rigorous justification for SD models and establish EKL as their natural information-theoretic foundation.
Paper Structure (25 sections, 12 theorems, 138 equations, 2 figures, 2 tables)

This paper contains 25 sections, 12 theorems, 138 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Expected Kullback-Leibler reducing updates
  3. Preliminaries
  4. Main result: Only SD updates are EKL reducing
  5. Extension to locally bounded Hessians in expectation
  6. Upper bounds for the learning-rate matrix
  7. Comparison to related performance criteria
  8. CEV of Gorgi2023
  9. MSE of Gorgi2023
  10. EGMM of Creal2024GMM
  11. TKL of blasques2015information
  12. Examples
  13. Applicability to eleven univariate model densities
  14. Bivariate dynamic parameter: Gaussian location-scale model
  15. Conclusion
  16. ...and 10 more sections

Key Result

Theorem 1

Consider a class ${\mathcal{P}}$ and let Assumptions ass:thmiffeklindepc and ass:HB hold. Then, for each $\vartheta_{t\vert t-1}\in \Theta$ and $p_t \in {\mathcal{P}}$ such that $\mathbb{E}_{p_t}\!\left[\Delta \phi(Y_t,\vartheta_{t\vert t-1})\right]^\top \mathbb{E}_{p_t}\![{s}(X_t,\vartheta_{t\vert

Figures (2)

  • Figure 1: SD location model in Example \ref{['exmpl:LKLProblem1']} with two hypothetical true densities $p_t$.
  • Figure H.1: Illustration of a score-driven conditional mean model with four hypothetical truths in the four panels.

Theorems & Definitions (32)

  • Definition 1: Score-driven update
  • Example 1
  • Definition 2: EKL difference
  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Proposition 1
  • Remark 1
  • Remark 2
  • Theorem 3
  • ...and 22 more