Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Time-Series Training Dataset through Lens of Spectrum in Deep State Space Models

Sekitoshi Kanai, Yasutoshi Ida, Kazuki Adachi, Mihiro Uchida, Tsukasa Yoshida, Shin'ya Yamaguchi

TL;DR

This work introduces the K-spectral metric to evaluate time-series training datasets for deep SSMs by examining the spectrum of intermediate signals within the SSM blocks. Grounded in optimal input design and persistency of excitation, the metric aggregates the top-$K$ frequency magnitudes of intermediate signals, offering an early indicator of downstream performance. Empirical results across system identification, classification, and forecasting demonstrate that the K-spectral metric correlates strongly with test performance, often outperforming dataset size and first-epoch loss, especially under biased data conditions. The approach enables more efficient data collection and dataset curation for time-series tasks using deep SSMs, with potential extensions to dataset optimization and active learning.

Abstract

This study investigates a method to evaluate time-series datasets in terms of the performance of deep neural networks (DNNs) with state space models (deep SSMs) trained on the dataset. SSMs have attracted attention as components inside DNNs to address time-series data. Since deep SSMs have powerful representation capacities, training datasets play a crucial role in solving a new task. However, the effectiveness of training datasets cannot be known until deep SSMs are actually trained on them. This can increase the cost of data collection for new tasks, as a trial-and-error process of data collection and time-consuming training are needed to achieve the necessary performance. To advance the practical use of deep SSMs, the metric of datasets to estimate the performance early in the training can be one key element. To this end, we introduce the concept of data evaluation methods used in system identification. In system identification of linear dynamical systems, the effectiveness of datasets is evaluated by using the spectrum of input signals. We introduce this concept to deep SSMs, which are nonlinear dynamical systems. We propose the K-spectral metric, which is the sum of the top-K spectra of signals inside deep SSMs, by focusing on the fact that each layer of a deep SSM can be regarded as a linear dynamical system. Our experiments show that the K-spectral metric has a large absolute value of the correlation coefficient with the performance and can be used to evaluate the quality of training datasets.

Evaluating Time-Series Training Dataset through Lens of Spectrum in Deep State Space Models

TL;DR

This work introduces the K-spectral metric to evaluate time-series training datasets for deep SSMs by examining the spectrum of intermediate signals within the SSM blocks. Grounded in optimal input design and persistency of excitation, the metric aggregates the top- frequency magnitudes of intermediate signals, offering an early indicator of downstream performance. Empirical results across system identification, classification, and forecasting demonstrate that the K-spectral metric correlates strongly with test performance, often outperforming dataset size and first-epoch loss, especially under biased data conditions. The approach enables more efficient data collection and dataset curation for time-series tasks using deep SSMs, with potential extensions to dataset optimization and active learning.

Abstract

This study investigates a method to evaluate time-series datasets in terms of the performance of deep neural networks (DNNs) with state space models (deep SSMs) trained on the dataset. SSMs have attracted attention as components inside DNNs to address time-series data. Since deep SSMs have powerful representation capacities, training datasets play a crucial role in solving a new task. However, the effectiveness of training datasets cannot be known until deep SSMs are actually trained on them. This can increase the cost of data collection for new tasks, as a trial-and-error process of data collection and time-consuming training are needed to achieve the necessary performance. To advance the practical use of deep SSMs, the metric of datasets to estimate the performance early in the training can be one key element. To this end, we introduce the concept of data evaluation methods used in system identification. In system identification of linear dynamical systems, the effectiveness of datasets is evaluated by using the spectrum of input signals. We introduce this concept to deep SSMs, which are nonlinear dynamical systems. We propose the K-spectral metric, which is the sum of the top-K spectra of signals inside deep SSMs, by focusing on the fact that each layer of a deep SSM can be regarded as a linear dynamical system. Our experiments show that the K-spectral metric has a large absolute value of the correlation coefficient with the performance and can be used to evaluate the quality of training datasets.
Paper Structure (30 sections, 1 theorem, 27 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 30 sections, 1 theorem, 27 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. State Space Models in DNNs
  4. Optimal Input Design
  5. Persistency of Excitation
  6. Related Work
  7. Proposed Metrics
  8. K-spectral Metric
  9. Relationship with Optimal Input Design
  10. Relationship with PE
  11. Implementation of K-spectral Metric
  12. Experiments
  13. Common setup
  14. Models
  15. Baseline Methods
  16. ...and 15 more sections

Key Result

Theorem 3.1

$R(\bar{\bm{\mathcal{\mathcal{U}}}}_T,K)$ in Eq. (prop) achieves the maximum value $R^*_K=\max_{\bar{\bm{\mathcal{U}}}_T}R(\bar{\bm{\mathcal{U}}}_T,K)$ if and only if $|\bar{\mathcal{U}}_i|=|\bar{\mathcal{U}}_j|$ for all $i,j\in \mathrm{topk}(\bar{\bm{\mathcal{U}}}_T,K)$, and $|\bar{\mathcal{U}}_i|=

Figures (4)

  • Figure 1: Overview of evaluation of a training dataset by K-spectral metric (right) compared with input design of system identification (left). K-spectral metric is a sum of top-K magnitudes of frequency components $|\bar{U}^l_s|$ of $u^l_t$ applied to SSMs.
  • Figure 2: K-spectral metrics for four multiple sinusoidal signals $u^i_t=\sum_s c^i_s \mathrm{sin}(2\pi st/T)$ for four settings $i=1,\dots,4$. \ref{['Met']} plots K-spectral metrics for these signals. \ref{['Sig1']}-\ref{['Sig4']} are the waveforms (top), $|\mathcal{U}_T|$ by FFT (middle), and sorted $|\mathcal{U}_T|$ of signals (bottom). When a signal has the flat spectrum for $K=12$ points, our metric achieves the highest.
  • Figure 3: $\rho$ of $\bar{R}$ against Epochs in training S5 on (i).
  • Figure 5: $|\rho|$ of $\bar{R}$ against Layers in training S5 on (i)+(ii).

Theorems & Definitions (2)

  • Definition 2.1: ljung1999system
  • Theorem 3.1