Latent Space Representation of Electricity Market Curves for Improved Prediction Efficiency

TL;DR

This paper tackles the challenge of predicting high-dimensional, multivariate electricity market curves by proposing a three-phase ML framework that preprocesses data, reduces dimensionality using PCA, kPCA, or UMAP, and predicts curves directly in latent space with RF, LSTM, and the novel TSMixer architecture. Across experiments on the MIBEL dataset, nonlinear latent representations from UMAP in 2–3 dimensions consistently yield the lowest prediction errors for both supply and demand curves, with TSMixer often achieving the best performance. The study highlights substantial accuracy gains over PCA-based approaches and demonstrates the first ML-based prediction of demand curves alongside supply curves, offering richer market insights. Practically, the approach improves prediction efficiency and could be extended to other electricity markets and price-prediction tasks, informing participants and policymakers about evolving market dynamics.

Abstract

This work presents a three-phase ML prediction framework designed to handle a high dimensionality and multivariate time series character of the electricity market curves. In the preprocessing phase, we transform the original data to achieve a unified structure and mitigate the effect of possible outliers. Further, to address the challenge of high dimensionality, we test three dimensionality reduction techniques (PCA, kPCA, UMAP). Finally, we predict supply and demand curves, once represented in a latent space, with a variety of machine learning methods (RF, LSTM, TSMixer). As our results on the MIBEL dataset show, a high dimensional structure of the market curves can be best handled by the nonlinear reduction technique UMAP. Regardless of the ML technique used for prediction, we achieved the lowest values for all considered precision metrics with a UMAP latent space representation in only two or three dimensions, even when compared to PCA and kPCA with five or six dimensions. Further, we demonstrate that the most promising machine learning technique to handle the complex structure of the electricity market curves is a novel TSMixer architecture. Finally, we fill the gap in the field of electricity market curves prediction literature: in addition to standard analysis on the supply side, we applied the ML framework and predicted demand curves too. We discussed the differences in the achieved results for these two types of curves.

Figures (5)

• Figure 1: Market curves from the training set after preprocessing step III for November 14, 2018, at 12:00 p.m. The original supply and demand curves are shown in black and grey, respectively. Preprocessed curves are displayed in dark red for supply and light red for demand.
• Figure 2: Standard deviations of demand and supply curves over time.
• Figure 3: Zoomed in market curve predictions for July 14, 2020, at 03:00 a.m. The real supply and demand curves are shown in black and grey, respectively, while the predicted curves are displayed in dark red for supply and light red for demand.
• Figure 4: Algorithms average runtime for training (blue), transformation (orange), and inverse transformation (green) for a single supply curve. Average values were calculated over 1000 runs. The enlarged rectangle shows a comparison between PCA and kPCA.
• Figure 5: Algorithms average runtime for training (blue), transformation (orange), and inverse transformation (green) for a single demand curve. Average values were calculated over 1000 runs. The enlarged rectangle shows a comparison between PCA and kPCA.