An Efficient and Explainable Transformer-Based Few-Shot Learning for Modeling Electricity Consumption Profiles Across Thousands of Domains

TL;DR

This paper tackles the challenge of modeling electricity consumption profiles (ECPs) under data scarcity across thousands of domains. It proposes a lightweight, explainable few-shot learning framework that uses a Transformer encoder to predict Gaussian Mixture Model (GMM) parameters for target domains from very limited samples, with a two-stage process: within-domain z-step EM and cross-domain residual transfer. The authors show that their method can accurately restore the target ECP distribution using as little as 1.6% of the complete domain data, outperforming a strong time-series imputation baseline while remaining interpretable through mean/variance shifts of GMM components. The approach is validated on a large, diverse dataset of individual household ECPs, and is positioned to scale to thousands of domains in practical energy systems, with open-source code and future work planned toward Copula-based extensions for richer dependency modeling.

Abstract

Electricity Consumption Profiles (ECPs) are crucial for operating and planning power distribution systems, especially with the increasing numbers of various low-carbon technologies such as solar panels and electric vehicles. Traditional ECP modeling methods typically assume the availability of sufficient ECP data. However, in practice, the accessibility of ECP data is limited due to privacy issues or the absence of metering devices. Few-shot learning (FSL) has emerged as a promising solution for ECP modeling in data-scarce scenarios. Nevertheless, standard FSL methods, such as those used for images, are unsuitable for ECP modeling because (1) these methods usually assume several source domains with sufficient data and several target domains. However, in the context of ECP modeling, there may be thousands of source domains with a moderate amount of data and thousands of target domains. (2) Standard FSL methods usually involve cumbersome knowledge transfer mechanisms, such as pre-training and fine-tuning, whereas ECP modeling requires more lightweight methods. (3) Deep learning models often lack explainability, hindering their application in industry. This paper proposes a novel FSL method that exploits Transformers and Gaussian Mixture Models (GMMs) for ECP modeling to address the above-described issues. Results show that our method can accurately restore the complex ECP distribution with a minimal amount of ECP data (e.g., only 1.6\% of the complete domain dataset) while it outperforms state-of-the-art time series modeling methods, maintaining the advantages of being both lightweight and interpretable. The project is open-sourced at https://github.com/xiaweijie1996/TransformerEM-GMM.git.

Figures (7)

• Figure 1: Modeling ECP distribution across many domains (households) using FSL. $L_s$ and $L_t$ are the numbers of source and target domains (households), respectively.
• Figure 2: Our proposed method begins with $\theta_o$ as the initial parameter of the GMMs. Let $\theta_r$ be the optimal parameter for the target domain (or the estimated parameter assuming a complete ECP dataset). After applying the $z$-step EM algorithm on limited target-domain data, we obtain the estimated parameters $\theta_e$. If the GMMs converge on this limited data, we achieve $\theta_p$. Our approach uses a Transformer to predict $\vec{er}$ such that $\theta_r = \vec{er} + \theta_e$.
• Figure 3: Training/inference process of one domain. $\hat{\theta_r} = \{ \mu_j, \sigma_j \}^J_{j=1}$ represents the predicted parameter of GMMs, $\mathbf{w}$ is the weights of components, $l(\cdot)$ is the loss function. In the training process, $\mathcal{D}^k \in \mathcal{S}$. In the inference process, $\mathcal{D}^k \in \mathcal{T}$, and only $\hat{\theta}_r$ is predicted without loss computation and parameter updating for the Transformer.
• Figure 4: $z$ step EM algorithm (Within-domain Tuning) aims at learning target-domain specific knowledge. $x^k_i$ is the input ECP samples of $k$-th domain and $i$-th sample, in this process. $J$ is the number of components of GMMs, each $\mu$ and $\sigma$ are the parameters of a Gaussian component in GMMs.
• Figure 5: Knowledge-transfer Tuning process. Sampled ECP samples are fed into the encoder together with corresponding $\theta_e$ to predict $\vec{er}$. $d(\cdot)$ represents the date embedding, which indicates the day of the year for $x_i$, and $p(\cdot)$ represents the parameter information, indicating which Gaussian component of the GMMs $\mu$ and $\sigma$ belong to. We use the [Pad] token to align the shapes of inputs with different amounts $n^{k}$.