On robustness of Spectral Rényi divergence

TL;DR

The paper develops and analyzes spectral $\alpha$-Rényi divergences for time-series spectral densities, showing Itakura–Saito is the $\alpha\to1$ limit and establishing a link to the $\gamma$-divergence via Szegő's limit theorem. It derives a variational representation and demonstrates that the minimum spectral Rényi divergence estimator enjoys robust, stable optimization paths in the presence of frequency-domain outliers, unlike the Itakura–Saito estimator. Theoretical results are complemented by simulations on Brune-type spectral models, illustrating robustness across $\alpha\in\{0.5,0.75,0.9\}$ and guiding practical choices for $\alpha$. Overall, the work provides both information-theoretic justification and computational robustness insights for using spectral $\alpha$-Rényi divergences in spectral density estimation.

Abstract

This paper studies a specific category of statistical divergences for spectral densities of time series: the spectral $α$-Rényi divergences, which includes the Itakura--Saito divergence as a subset. The aim of this paper is to highlight both information-theoretic and statistical properties of spectral $α$-Rényi divergences. We reveal the connection between the spectral $α$-Rényi divergence and the $γ$-divergence in robust statistics, and a variational representation of spectral $α$-Rényi divergence. Inspired by these results suggesting ``robustness'' of spectral $α$-Rényi divergence, we show that the minimum spectral Rényi divergence estimate has a stable optimization path with respect to outliers in the frequency domain, unlike the minimum Itakura-Saito divergence estimator, and thus it delivers more stable estimate, reducing the need for intricate pre-processing.

Figures (5)

• Figure 1: Comparison of optimization paths. The optimization paths of the gradient descent for the $AR(1)$ model, with respect to $(\log \sigma, 2\{1/(1+\exp(-\rho))-1/2\})$, are presented under the spectral Rényi divergence minimization and the Itakura--Saito divergence minimization. The left figure shows the optimization paths (curves with light colors) of the spectral Rényi divergence minimization that start from the initial value (the gray circle) and terminate at points after $5000$ iterations (the inverted triangles), where the colors indicates different values of the contamination $z$. The right figure shows the corresponding results for the Itakura--Saito divergence minimization.
• Figure 2: Spectral densities with the estimates plugged-in without any trend. The gray curve is the periodogram. The true spectral density is colored in black. Spectral densities based on the spectral Rényi divergence ($\alpha=0.5,0.75,0.9$) are colored in red, salmon pink, and green,respectively. The spectral density based on the Itakura--Saito divergence is colored in blue. (a) the result based on the initial value $\theta^{(0),1}$, (b) the result based on the initial value $\theta^{(0),2}$, (c) the result based on the initial value $\theta^{(0),3}$.
• Figure 3: Spectral densities with the estimates plugged-in with the trigonometric trends. The red dashed lines denote the frequencies at which the outliers are injected. The gray curve is the periodogram. The true spectral density is colored in black. Spectral densities based on the spectral Rényi divergence ($\alpha=0.5,0.75,0.9$) are colored in red, salmon pink, and green,respectively. The spectral density based on the Itakura--Saito divergence is colored in blue. (a) the result based on the initial value $\theta^{(0),1}$, (b) the result based on the initial value $\theta^{(0),2}$, (c) the result based on the initial value $\theta^{(0),3}$.
• Figure 4: Spectral densities with the estimates plugged-in for the Brune spectral model with attenuation and with the trigonometric trends of $z_{1}=z_{2}=2.5$. The gray curve is the periodogram. The true spectral density is colored in black. Spectral densities based on the spectral Rényi divergence ($\alpha=0.5,0.75,0.9$) are colored in red, salmon pink, and green,respectively. The spectral density based on the Itakura--Saito divergence is colored in blue. The red dashed lines denote the frequencies at which the outliers are injected. (a) the result based on the initial value $\theta^{(0),1}$, (b) the result based on the initial value $\theta^{(0),2}$, (c) the result based on the initial value $\theta^{(0),3}$.
• Figure 5: Spectral densities with the estimates plugged-in for the Brune spectral model with attenuation and with the trigonometric trends of $z_{1}=z_{2}=25$. The gray curve is the periodogram. The true spectral density is colored in black. Spectral densities based on the spectral Rényi divergence ($\alpha=0.5,0.75,0.9$) are colored in red, salmon pink, and green,respectively. The spectral density based on the Itakura--Saito divergence is colored in blue. The red dashed lines denote the frequencies at which the outliers are injected. (a) the result based on the initial value $\theta^{(0),1}$, (b) the result based on the initial value $\theta^{(0),2}$, (c) the result based on the initial value $\theta^{(0),3}$.

Theorems & Definitions (9)

• Theorem 1: The $\gamma$-divergence leads to spectral Rényi divergences
• Theorem 2: Variational representation of the spectral Rényi divergence
• Proposition 1
• Theorem 3: Stability of optimization path with a fixed learning rate sequence
• Remark 1: Constant step size
• Remark 2: AR(1) model
• Proposition 2: Stability of line search
• Remark 3: Stability of optimization path with the Armijo condition
• Proposition 3