Automatic selection of the best neural architecture for time series forecasting via multi-objective optimization and Pareto optimality conditions

TL;DR

The paper introduces a flexible, automated framework to design time-series forecasting models by composing GRU, LSTM, multi-head Attention, and SSM blocks into parameterized architectures. It formulates a multi-objective optimization problem over architectural choices, producing Pareto-optimal designs with objectives including relative $L_2$ error, training time, and parameter count, then selects the best design via a user-defined preference and normalizes comparisons with $r$ to obtain $\hat f$. Across four real-world applications (biomedical glucose forecasting, ocean-wave forecasting, VIV of marine risers, and floating offshore structure motions), the study shows no single architecture dominates; optimal designs are context-dependent and often involve composite block configurations. The results highlight the value of Pareto-based design and preference-driven selection for tailoring architectures to specific accuracy and efficiency requirements, while LP-based rediscovery provides a scalable way to infer the preference weights that underlie observed optima. The work also notes substantial offline training costs and suggests efficient sampling strategies to accelerate Pareto-front discovery in real-time settings.

Abstract

Time series forecasting plays a pivotal role in a wide range of applications, including weather prediction, healthcare, structural health monitoring, predictive maintenance, energy systems, and financial markets. While models such as LSTM, GRU, Transformers, and State-Space Models (SSMs) have become standard tools in this domain, selecting the optimal architecture remains a challenge. Performance comparisons often depend on evaluation metrics and the datasets under analysis, making the choice of a universally optimal model controversial. In this work, we introduce a flexible automated framework for time series forecasting that systematically designs and evaluates diverse network architectures by integrating LSTM, GRU, multi-head Attention, and SSM blocks. Using a multi-objective optimization approach, our framework determines the number, sequence, and combination of blocks to align with specific requirements and evaluation objectives. From the resulting Pareto-optimal architectures, the best model for a given context is selected via a user-defined preference function. We validate our framework across four distinct real-world applications. Results show that a single-layer GRU or LSTM is usually optimal when minimizing training time alone. However, when maximizing accuracy or balancing multiple objectives, the best architectures are often composite designs incorporating multiple block types in specific configurations. By employing a weighted preference function, users can resolve trade-offs between objectives, revealing novel, context-specific optimal architectures. Our findings underscore that no single neural architecture is universally optimal for time series forecasting. Instead, the best-performing model emerges as a data-driven composite architecture tailored to user-defined criteria and evaluation objectives.

Figures (8)

• Figure 1: A schematic representation of the entire architecture (a) of the composite model with details on the type of blocks, e.g., the attention mechanism in (b) and the SSM module in (c).
• Figure 2: First Application: (a) Pareto front evaluating relative error--training time--number of parameters, and (b) discovery of optimal architectures using different preference functions for the glucose prediction.
• Figure 3: Second Application: (a) Pareto front evaluating relative error--training time--number of parameters, and (b) discovery of optimal architectures using different preference functions, for significant wave heights.
• Figure 4: Third Application: (a) Pareto front evaluating relative error--training time--number of parameters, and (b) discovery of optimal architectures using different preference functions, for VIV of the marine risers.
• Figure 5: Third Application: Designed network architectures for marine risers based on given preference functions: (a) $p^1$, (b) $p^2$, (c) $p^3$, (d) $p^4$, (e) $p^5$, (f) $p^6$, (g) $p^7$.

Theorems & Definitions (5)

• Definition 1
• Definition 2
• Lemma 1: See lee2024automatic
• Remark 1
• Definition 3