Time-series Random Process Complexity Ranking Using a Bound on Conditional Differential Entropy

TL;DR

This work addresses ranking the intrinsic complexity of time-series when the underlying distribution is unknown. It builds a tractable framework by leveraging conditional differential entropy bounds that relate uncertainty in the next observation to the covariance of prediction errors, and extends these bounds with Hadamard's inequality to tighten the upper limit. The authors introduce PECEP, a prediction-error-based proxy for conditional entropy, and validate it through controlled VAR experiments with known entropy and a bio-inspired audio ranking task where entropy is unknown. The results show that PECEP tracks theoretical entropy bounds as data grows and that neural predictors can recover meaningful complexity ordering, offering a practical approach for comparing temporal dynamics in real-world, high-dimensional time series.

Abstract

Conditional differential entropy provides an intuitive measure for relatively ranking time-series complexity by quantifying uncertainty in future observations given past context. However, its direct computation for high-dimensional processes from unknown distributions is often intractable. This paper builds on the information theoretic prediction error bounds established by Fang et al. \cite{fang2019generic}, which demonstrate that the conditional differential entropy \textbf{$h(X_k \mid X_{k-1},...,X_{k-m})$} is upper bounded by a function of the determinant of the covariance matrix of next-step prediction errors for any next step prediction model. We add to this theoretical framework by further increasing this bound by leveraging Hadamard's inequality and the positive semi-definite property of covariance matrices. To see if these bounds can be used to rank the complexity of time series, we conducted two synthetic experiments: (1) controlled linear autoregressive processes with additive Gaussian noise, where we compare ordinary least squares prediction error entropy proxies to the true entropies of various additive noises, and (2) a complexity ranking task of bio-inspired synthetic audio data with unknown entropy, where neural network prediction errors are used to recover the known complexity ordering. This framework provides a computationally tractable method for time-series complexity ranking using prediction errors from next-step prediction models, that maintains a theoretical foundation in information theory.

Figures (4)

• Figure 1: Experiment 1 Setup. The top panel shows the ground truth signal across 32 feature dimensions over time. The middle panel shows the time-aligned predictions made by Ordinary Least Squares (OLS). The bottom panel shows the residual errors (ground truth minus OLS prediction) used for calculating the Prediction Error Conditional Entropy Proxy (PECEP).
• Figure 2: Experiment 2 setup. The top row contains example spectrograms of the simplest species 0 and most complex species 9. Species 0 is made up of calls with a smaller frequency range, less variance in power between spectrogram columns containing calls, and less variance between vocalizations. The middle row shows the spectrogram column predictions of a fully connected neural network. The bottom row shows the difference between the ground truth top row and the predicted middle row. The green vertical lines indicate the onset-offsets of the calls as we collect the PECEP scores from the residuals of just those audio segments.
• Figure 3: Experiment 1 Results. PECEP Convergence Validation on Synthetic Autoregressive (AR) Data. Mean PECEP values across 30 trials for the ordinary least-squares AR coefficient model and an Oracle that already knows the ground-truth coefficients at four additive Gaussian noise levels $\mathcal{N}(0,\sigma^2I); \sigma^2 \in \{0.001, 0.01, 0.1, 1.0\}$ and nine dataset sizes. Dashed lines indicate theoretical conditional differential entropy bounds.
• Figure 4: Experiment 2 Results. Prediction Error Conditional Entropy (PECEP) boxplots generated from the prediction errors on artificial calls from synthetic species. As the species monotonically grow in acoustic complexity, so do the median PECEP values.