Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stable Training of Probabilistic Models Using the Leave-One-Out Maximum Log-Likelihood Objective

Kutay Bölat, Simon H. Tindemans, Peter Palensky

TL;DR

This paper addresses robust probabilistic modelling of power-system data when historical data are limited. It identifies data-copying as a singularity in adaptive KDE under the standard MLL objective and proposes a Leave-One-Out MLL criterion to prevent it, providing theoretical guarantees for non-singular solutions. It further extends adaptive KDE with kernel weights ($\pi$-KDE) and develops a modified EM algorithm to maximize the LOO objective reliably. Through experiments on Europe and Denmark datasets, the authors show that $\pi$-KDE offers strong estimation performance while preserving robustness against singularities, outperforming adaptive KDE and comparable GMM baselines in practice. The work enables more reliable, data-efficient probabilistic modelling for power systems, with potential benefits for edge computing and tail-event analysis.

Abstract

Probabilistic modelling of power systems operation and planning processes depends on data-driven methods, which require sufficiently large datasets. When historical data lacks this, it is desired to model the underlying data generation mechanism as a probability distribution to assess the data quality and generate more data, if needed. Kernel density estimation (KDE) based models are popular choices for this task, but they fail to adapt to data regions with varying densities. In this paper, an adaptive KDE model is employed to circumvent this, where each kernel in the model has an individual bandwidth. The leave-one-out maximum log-likelihood (LOO-MLL) criterion is proposed to prevent the singular solutions that the regular MLL criterion gives rise to, and it is proven that LOO-MLL prevents these. Relying on this guaranteed robustness, the model is extended by adjustable weights for the kernels. In addition, a modified expectation-maximization algorithm is employed to accelerate the optimization speed reliably. The performance of the proposed method and models are exhibited on two power systems datasets using different statistical tests and by comparison with Gaussian mixture models. Results show that the proposed models have promising performance, in addition to their singularity prevention guarantees.

Stable Training of Probabilistic Models Using the Leave-One-Out Maximum Log-Likelihood Objective

TL;DR

This paper addresses robust probabilistic modelling of power-system data when historical data are limited. It identifies data-copying as a singularity in adaptive KDE under the standard MLL objective and proposes a Leave-One-Out MLL criterion to prevent it, providing theoretical guarantees for non-singular solutions. It further extends adaptive KDE with kernel weights (-KDE) and develops a modified EM algorithm to maximize the LOO objective reliably. Through experiments on Europe and Denmark datasets, the authors show that -KDE offers strong estimation performance while preserving robustness against singularities, outperforming adaptive KDE and comparable GMM baselines in practice. The work enables more reliable, data-efficient probabilistic modelling for power systems, with potential benefits for edge computing and tail-event analysis.

Abstract

Probabilistic modelling of power systems operation and planning processes depends on data-driven methods, which require sufficiently large datasets. When historical data lacks this, it is desired to model the underlying data generation mechanism as a probability distribution to assess the data quality and generate more data, if needed. Kernel density estimation (KDE) based models are popular choices for this task, but they fail to adapt to data regions with varying densities. In this paper, an adaptive KDE model is employed to circumvent this, where each kernel in the model has an individual bandwidth. The leave-one-out maximum log-likelihood (LOO-MLL) criterion is proposed to prevent the singular solutions that the regular MLL criterion gives rise to, and it is proven that LOO-MLL prevents these. Relying on this guaranteed robustness, the model is extended by adjustable weights for the kernels. In addition, a modified expectation-maximization algorithm is employed to accelerate the optimization speed reliably. The performance of the proposed method and models are exhibited on two power systems datasets using different statistical tests and by comparison with Gaussian mixture models. Results show that the proposed models have promising performance, in addition to their singularity prevention guarantees.
Paper Structure (33 sections, 4 theorems, 17 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 33 sections, 4 theorems, 17 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Motivating Examples
  4. Synthetic data generation
  5. Rare-event sampling
  6. Preliminaries
  7. Maximum Log-likelihood Criterion
  8. Kernel Density Estimation
  9. Adaptive Kernel Density Estimation
  10. Data-copying as a Singularity
  11. Methodology
  12. Leave-One-Out Maximum Log-Likelihood Objective for Adaptive Kernel Density Estimation
  13. $\pi$-Kernel Density Estimation
  14. Modified Expectation-Maximization Algorithm
  15. Experimentation
  16. ...and 18 more sections

Key Result

Theorem 1

An A-KDE model optimized by MLL objective copies at least one data point if and only if the total log-likelihood goes to infinity, i.e.

Figures (4)

  • Figure 1: Overall objective of probabilistic modelling. Objective functions (MLL, LOO-MLL) can only use the data points. MLL directly targets the dataset, and it copies a random data point (singularity). LOO-MLL avoids this, and the resulting model is similar to the real target distribution.
  • Figure 2: Visual comparison between the training and the generated daily consumption data for three countries (NO, FR, IT) from the Europe dataset (Section \ref{['subs:datasets']}), illustrating complex dependencies.
  • Figure 3: Convergence times of the modified EM and Adam.
  • Figure 4: ECDFs of sample test scores for (a) Europe and (b) Denmark datasets.

Theorems & Definitions (8)

  • Definition 1: Data-copying
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • proof
  • Corollary 3.1
  • proof
  • proof