A Meta-Knowledge-Augmented LLM Framework for Hyperparameter Optimization in Time-Series Forecasting

TL;DR

Hyperparameter optimization for time-series forecasting is hindered by computational expense and limited interpretability. The authors introduce LLM-AutoOpt, a forecasting-specific meta-knowledge–augmented HPO framework that couples Bayesian optimization for efficient initialization with structured LLM reasoning over data, model, and optimization history. The two-phase workflow constructs forecasting meta-knowledge and then iteratively refines hyperparameters via an LLM, producing context-aware and interpretable updates. Empirical results on the Jena Climate dataset show lower RMSE and more transparent optimization dynamics compared to BO and NoMeta baselines, albeit with higher computational overhead. The work demonstrates the value of explicit meta-knowledge in guiding LLMs for HPO in forecasting and points to broader validation and automated feature selection as avenues for future research.

Abstract

Hyperparameter optimization (HPO) plays a central role in the performance of deep learning models, yet remains computationally expensive and difficult to interpret, particularly for time-series forecasting. While Bayesian Optimization (BO) is a standard approach, it typically treats tuning tasks independently and provides limited insight into its decisions. Recent advances in large language models (LLMs) offer new opportunities to incorporate structured prior knowledge and reasoning into optimization pipelines. We introduce LLM-AutoOpt, a hybrid HPO framework that combines BO with LLM-based contextual reasoning. The framework encodes dataset meta-features, model descriptions, historical optimization outcomes, and target objectives as structured meta-knowledge within LLM prompts, using BO to initialize the search and mitigate cold-start effects. This design enables context-aware and stable hyperparameter refinement while exposing the reasoning behind optimization decisions. Experiments on a multivariate time series forecasting benchmark demonstrate that LLM-AutoOpt achieves improved predictive performance and more interpretable optimization behavior compared to BO and LLM baselines without meta-knowledge.

Figures (4)

• Figure 1: Automated hyperparameter optimization guided by LLM reasoning.
• Figure 2: Validation RMSE convergence across optimization strategies
• Figure 3: Relative humidity (RH %) forecast comparison across different models
• Figure 4: Comparison of reasoning outputs for hyperparameter recommendation with and without meta-knowledge injection.