Horizon Activation Mapping for Neural Networks in Time Series Forecasting

TL;DR

Horizon Activation Mapping (HAM) provides a cross-family, gradient-based interpretability framework for time-series forecasting that abstracts away specific architectural choices. By using causal and anti-causal horizon masks, HAM analyzes gradient-update magnitudes across horizon subseries and yields Ratess of Change, Gradient Equivariant Points, Interpolated Area Plots, and Difference Plots to compare learning dynamics. The study applies HAM to diverse model families (e.g., NHITS, FEDformer, Pyraformer, SpaceTime, Multi-Resolution DDPM) on the ETTm2 dataset, across horizon sizes up to $H=720$ and under varying optimization settings (dropouts, batch sizes, early stopping, splits), uncovering model-specific and training-condition–dependent patterns such as polynomial batch-size effects and exponent-like trends in certain architectures. These insights facilitate granular model selection, validation-set design, and cross-family comparisons for horizon-aware forecasting, enabling more informed architectural choices and training strategies.

Abstract

Neural networks for time series forecasting have relied on error metrics and architecture-specific interpretability approaches for model selection that don't apply across models of different families. To interpret forecasting models agnostic to the types of layers across state-of-the-art model families, we introduce Horizon Activation Mapping (HAM), a visual interpretability technique inspired by grad-CAM that uses gradient norm averages to study the horizon's subseries where grad-CAM studies attention maps over image data. We introduce causal and anti-causal modes to calculate gradient update norm averages across subseries at every timestep and lines of proportionality signifying uniform distributions of the norm averages. Optimization landscape studies with respect to changes in batch sizes, early stopping, train-val-test splits, univariate forecasting and dropouts are studied with respect to performances and subseries in HAM. Interestingly, batch size based differences in activities seem to indicate potential for existence of an exponential approximation across them per epoch relative to each other. Multivariate forecasting models including MLP-based CycleNet, N-Linear, N-HITS, self attention-based FEDformer, Pyraformer, SSM-based SpaceTime and diffusion-based Multi-Resolution DDPM over different horizon sizes trained over the ETTm2 dataset are used for HAM plots in this study. NHITS' neural approximation theorem and SpaceTime's exponential autoregressive activities have been attributed to trends in HAM plots over their training, validation and test sets. In general, HAM can be used for granular model selection, validation set choices and comparisons across different neural network model families.

Figures (9)

• Figure 1: $H=720$NHITS gradient activities with and without a dropout of 0.2 chosen based on performance.
• Figure 2: HAMs of $H=720$NHITS models with different batch sizes. The plots get darker corresponding to increases in batch sizes.
• Figure 3: Although different in magnitudes when batch sizes change, over converged N-Linear, NHITS and CycleNet models, gradient norm averages interpolated by batch size show a linearly increasing scale $S$ across subseries and a polynomially saturating trend as batch sizes increase. This suggests that HAM plots could find polynomial approximations across batch sizes one epoch at a time.
• Figure 4: $H=720$CycleNet models with different batch sizes show norm averages increasing in the number of epochs. The numbers next to CycleNet correspond to epochs with 0 indicating the randomly initialized model's. The plots are darker corresponding to increases in epochs.
• Figure 5: $H=720$NHITS models after early stopping show the reduction in norm average magnitudes away from the local minima. The plots get darker corresponding to increases in epoch counts.