Differential Evolution Algorithm based Hyper-Parameters Selection of Transformer Neural Network Model for Load Forecasting

TL;DR

This work tackles short-term load forecasting by employing an encoder-only Transformer whose hyperparameters are optimized with Differential Evolution. The approach is benchmarked against GA and PSO, using MSE as the optimization objective and MAPE as the accuracy metric on a Canadian Ottawa dataset. Results show Differential Evolution yields the lowest MAPE (1.11) compared with GA (1.31), PSO (1.28), and manual tuning, demonstrating the method's effectiveness in hyperparameter search for nonlinear, nonconvex forecasting tasks. The study highlights the practical potential of metaheuristic-enhanced Transformer models for accurate energy demand prediction and outlines future work on larger populations and additional metaheuristics.

Abstract

Accurate load forecasting plays a vital role in numerous sectors, but accurately capturing the complex dynamics of dynamic power systems remains a challenge for traditional statistical models. For these reasons, time-series models (ARIMA) and deep-learning models (ANN, LSTM, GRU, etc.) are commonly deployed and often experience higher success. In this paper, we analyze the efficacy of the recently developed Transformer-based Neural Network model in Load forecasting. Transformer models have the potential to improve Load forecasting because of their ability to learn long-range dependencies derived from their Attention Mechanism. We apply several metaheuristics namely Differential Evolution to find the optimal hyperparameters of the Transformer-based Neural Network to produce accurate forecasts. Differential Evolution provides scalable, robust, global solutions to non-differentiable, multi-objective, or constrained optimization problems. Our work compares the proposed Transformer based Neural Network model integrated with different metaheuristic algorithms by their performance in Load forecasting based on numerical metrics such as Mean Squared Error (MSE) and Mean Absolute Percentage Error (MAPE). Our findings demonstrate the potential of metaheuristic-enhanced Transformer-based Neural Network models in Load forecasting accuracy and provide optimal hyperparameters for each model.

Figures (6)

• Figure 1: Differential Evolution Algorithm
• Figure 2: Mechanism of the Differential algorithm based hyper-parameters selection approach for the Load Forecasting task.
• Figure 3: The proposed Transformer-based deep learning model
• Figure 4: Training & Validation Loss vs Epochs plots for the Transformer-based Neural Network DE model
• Figure 5: Predicted plots for hourly demand for next 24 hours starting from the $N$-th hour for Transformer-based Neural Network DE Model.