Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a Deeper Understanding of Transformer for Residential Non-intrusive Load Monitoring

Minhajur Rahman, Yasir Arafat

TL;DR

This paper investigates how transformer architectures perform in residential non-intrusive load monitoring (NILM) by conducting a comprehensive hyper-parameter study on aspects such as hidden dimensions, number of layers, attention heads, dropout, and BERT-style masking. Using the REDD dataset, it shows that a compact transformer with a carefully chosen configuration can outperform prior NILM transformers like BERT4NILM while drastically reducing parameter count and improving training efficiency. The study provides concrete guidance on architectural choices and training strategies to optimize NILM performance, offering a foundation for scalable and robust transformer-based energy disaggregation. Overall, the work advances practical guidelines for deploying efficient transformer models in NILM and highlights potential for broader residential energy management applications.

Abstract

Transformer models have demonstrated impressive performance in Non-Intrusive Load Monitoring (NILM) applications in recent years. Despite their success, existing studies have not thoroughly examined the impact of various hyper-parameters on model performance, which is crucial for advancing high-performing transformer models. In this work, a comprehensive series of experiments have been conducted to analyze the influence of these hyper-parameters in the context of residential NILM. This study delves into the effects of the number of hidden dimensions in the attention layer, the number of attention layers, the number of attention heads, and the dropout ratio on transformer performance. Furthermore, the role of the masking ratio has explored in BERT-style transformer training, providing a detailed investigation into its impact on NILM tasks. Based on these experiments, the optimal hyper-parameters have been selected and used them to train a transformer model, which surpasses the performance of existing models. The experimental findings offer valuable insights and guidelines for optimizing transformer architectures, aiming to enhance their effectiveness and efficiency in NILM applications. It is expected that this work will serve as a foundation for future research and development of more robust and capable transformer models for NILM.

Towards a Deeper Understanding of Transformer for Residential Non-intrusive Load Monitoring

TL;DR

This paper investigates how transformer architectures perform in residential non-intrusive load monitoring (NILM) by conducting a comprehensive hyper-parameter study on aspects such as hidden dimensions, number of layers, attention heads, dropout, and BERT-style masking. Using the REDD dataset, it shows that a compact transformer with a carefully chosen configuration can outperform prior NILM transformers like BERT4NILM while drastically reducing parameter count and improving training efficiency. The study provides concrete guidance on architectural choices and training strategies to optimize NILM performance, offering a foundation for scalable and robust transformer-based energy disaggregation. Overall, the work advances practical guidelines for deploying efficient transformer models in NILM and highlights potential for broader residential energy management applications.

Abstract

Transformer models have demonstrated impressive performance in Non-Intrusive Load Monitoring (NILM) applications in recent years. Despite their success, existing studies have not thoroughly examined the impact of various hyper-parameters on model performance, which is crucial for advancing high-performing transformer models. In this work, a comprehensive series of experiments have been conducted to analyze the influence of these hyper-parameters in the context of residential NILM. This study delves into the effects of the number of hidden dimensions in the attention layer, the number of attention layers, the number of attention heads, and the dropout ratio on transformer performance. Furthermore, the role of the masking ratio has explored in BERT-style transformer training, providing a detailed investigation into its impact on NILM tasks. Based on these experiments, the optimal hyper-parameters have been selected and used them to train a transformer model, which surpasses the performance of existing models. The experimental findings offer valuable insights and guidelines for optimizing transformer architectures, aiming to enhance their effectiveness and efficiency in NILM applications. It is expected that this work will serve as a foundation for future research and development of more robust and capable transformer models for NILM.
Paper Structure (19 sections, 11 equations, 4 figures, 1 table)

This paper contains 19 sections, 11 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related works
  3. Our contribution
  4. Preliminaries
  5. Single-head self-attention mechanism
  6. Multi-head self-attention mechanism
  7. Problem Formulation
  8. Methodology
  9. System Architecture
  10. Proposed transformer architecture
  11. Experimental Settings and Results
  12. Experimental settings and baselines
  13. Analysis of results
  14. Evaluation of various number of layers
  15. Evaluation of various numbers of attention heads
  16. ...and 4 more sections

Figures (4)

  • Figure 1: An overview of NILM in a residential unit. Given an aggregated load of a household, it can decompose into individual appliance-level loads for understanding energy usage and further regulation or load monitoring.
  • Figure 2: NILM system architecture used in this paper. Best viewed in Zoom.
  • Figure 3: Transformer architecture used in this paper. Given an aggregated load, it predicts the load of a target appliance. Best viewed in Zoom.
  • Figure 4: Computation time/epoch (in secs). Max. value in Table \ref{['tab: results']} is compared.