Relative Geometry of Neural Forecasters: Linking Accuracy and Alignment in Learned Latent Geometry

TL;DR

This work study neural forecasters through the lens of representational alignment, introducing anchor-based, geometry-agnostic relative embeddings that remove rotational and scaling ambiguities in latent spaces.

Abstract

Neural networks can accurately forecast complex dynamical systems, yet how they internally represent underlying latent geometry remains poorly understood. We study neural forecasters through the lens of representational alignment, introducing anchor-based, geometry-agnostic relative embeddings that remove rotational and scaling ambiguities in latent spaces. Applying this framework across seven canonical dynamical systems - ranging from periodic to chaotic - we reveal reproducible family-level structure: multilayer perceptrons align with other MLPs, recurrent networks with RNNs, while transformers and echo-state networks achieve strong forecasts despite weaker alignment. Alignment generally correlates with forecasting accuracy, yet high accuracy can coexist with low alignment. Relative geometry thus provides a simple, reproducible foundation for comparing how model families internalize and represent dynamical structure.

Figures (15)

• Figure 1: Relative embeddings reveal consistent geometric structure across model families while removing rotational and scaling ambiguities. (a) Encoder–propagator–decoder forecasters take an input window of $L$ past states $\mathbf x_{t-L+1:t}$, embed it into a latent vector $\mathbf z$, and decode a prediction of the next $H$ states $\widehat{\mathbf x}_{t+1:t+H}$. To compare different forecasters, we compute absolute latent embeddings from data, transform them into anchor-based relative embeddings using moschella2023relative, and quantify alignment between forecasters using representational similarity scores. (b) Alignment–performance endpoints after training for RNNs (blue) and MLPs (green). RNNs achieve higher representational similarity and prediction accuracy (MSE), while MLPs show a clearer correlation between alignment and performance across seeds. (c-f) Example systems: Lorenz–63 (c), double pendulum (d), random skew (e), limit cycle (f). Columns display system trajectories, absolute embeddings (PCA; two or three principal components depending on dimensionality), relative embeddings (PCA), cross-forecaster similarity heatmaps averaged over five seeds—ordered as True System, MLP, Koopman MLP, NODE MLP, RNN, Autoregressive RNN, Koopman RNN, NODE RNN, Transformer, NODE Transformer, Koopman Transformer, and ESN; —and alignment–performance scatter plots across hyperparameter settings. Additional systems are shown in Appendix Figure \ref{['fig:main-fig-appendix']} and \ref{['fig:logistic_maps_heats']}.
• Figure 2: Performance–alignment trade-offs across training, noise, and input conditions. Columns show (a-m) training time evolution, (b-n) effects of input noise, (c-o) effects of sequence length $L$, and (d-p) test performance across model families (MLP, K-MLP: Koopman MLP, N-MLP: NODE MLP, RNN, A-RNN: Autoregressive RNN, K-RNN: Koopman RNN, N-RNN: NODE RNN, TF: Transformer, N-TF: NODE Transformer, K-TF: Koopman Transformer, ESN). Each point represents the mean squared error (MSE) and the representational similarity score (RSS) of a given a forecaster trained with a different random seed (color-coded by forecaster family; same-colored lines/points denote different initializations of the same forecaster). MLPs and RNNs exhibit consistent performance–alignment relationships, while transformers show larger variability; ESNs are excluded due to their no–backpropagation-through-time (no-BPTT) training. Increasing input noise consistently degrades both alignment and accuracy, whereas varying $L$ produces system-dependent effects, highlighting differences in robustness across model families. Test results (d-p) indicate that no single family dominates across all dynamical systems. Results for additional systems in Appendix \ref{['fig:perturbations2']}.
• Figure 3: Temporal evolution of representational alignment across dynamical systems. Each row shows the true system (left), reconstructed trajectories from different model families (MLPs, RNNs, transformers, ESN; same colour coding as in Figure \ref{['fig:perturbations']}), and their temporal similarity profiles (right; line thickness encodes time, $T=300$). For visualisation purposes we use relative coordinates $z'_i$ with respect to three anchor points (axes $z'_1$, $z'_2$, $z'_3$). For the Lorenz–63, double pendulum, and random skew systems, MLPs and RNNs maintain representations closely aligned with the true dynamics, whereas transformers and ESNs diverge. For the limit cycle and other periodic systems (see Appendix \ref{['fig:temporal_trajs2']}), all families capture similarly structured representations. The Logistic map is omitted due to its one-dimensional, contractive behavior.
• Figure 4: Forecasting and representational alignment. (a, b) Example systems: proper orthogonal decomposition (POD)-wake (a), Hopf (b). Columns show time series trajectories, absolute embeddings (visualized with principal component analysis (PCA); we plot the first 2 components for 2-dimensional systems and 3 components for the rest of the systems), relative embeddings (PCA), cross-forecaster similarity heatmaps (averaged over five seeds) with the order of True System, MLP, Koopman MLP, NODE MLP, RNN, Autoregressive RNN, Koopman RNN, NODE RNN, Transformer, NODE Transformer, Koopman Transformer, and ESN; and alignment–performance of forecasters with different hyperparameter settings.
• Figure 5: Cross-model similarity in Logistic Maps.