Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Distillation for Efficient Transformer-Based Reinforcement Learning in Hardware-Constrained Energy Management Systems

Pascal Henrich, Jonas Sievers, Maximilian Beichter, Thomas Blank, Ralf Mikut, Veit Hagenmeyer

Abstract

Transformer-based reinforcement learning has emerged as a strong candidate for sequential control in residential energy management. In particular, the Decision Transformer can learn effective battery dispatch policies from historical data, thereby increasing photovoltaic self-consumption and reducing electricity costs. However, transformer models are typically too computationally demanding for deployment on resource-constrained residential controllers, where memory and latency constraints are critical. This paper investigates knowledge distillation to transfer the decision-making behaviour of high-capacity Decision Transformer policies to compact models that are more suitable for embedded deployment. Using the Ausgrid dataset, we train teacher models in an offline sequence-based Decision Transformer framework on heterogeneous multi-building data. We then distil smaller student models by matching the teachers' actions, thereby preserving control quality while reducing model size. Across a broad set of teacher-student configurations, distillation largely preserves control performance and even yields small improvements of up to 1%, while reducing the parameter count by up to 96%, the inference memory by up to 90%, and the inference time by up to 63%. Beyond these compression effects, comparable cost improvements are also observed when distilling into a student model of identical architectural capacity. Overall, our results show that knowledge distillation makes Decision Transformer control more applicable for residential energy management on resource-limited hardware.

Knowledge Distillation for Efficient Transformer-Based Reinforcement Learning in Hardware-Constrained Energy Management Systems

Abstract

Transformer-based reinforcement learning has emerged as a strong candidate for sequential control in residential energy management. In particular, the Decision Transformer can learn effective battery dispatch policies from historical data, thereby increasing photovoltaic self-consumption and reducing electricity costs. However, transformer models are typically too computationally demanding for deployment on resource-constrained residential controllers, where memory and latency constraints are critical. This paper investigates knowledge distillation to transfer the decision-making behaviour of high-capacity Decision Transformer policies to compact models that are more suitable for embedded deployment. Using the Ausgrid dataset, we train teacher models in an offline sequence-based Decision Transformer framework on heterogeneous multi-building data. We then distil smaller student models by matching the teachers' actions, thereby preserving control quality while reducing model size. Across a broad set of teacher-student configurations, distillation largely preserves control performance and even yields small improvements of up to 1%, while reducing the parameter count by up to 96%, the inference memory by up to 90%, and the inference time by up to 63%. Beyond these compression effects, comparable cost improvements are also observed when distilling into a student model of identical architectural capacity. Overall, our results show that knowledge distillation makes Decision Transformer control more applicable for residential energy management on resource-limited hardware.

Paper Structure

This paper contains 26 sections, 12 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation in Energy
  4. Knowledge Distillation and Decision Transformer
  5. Decision Transformer in Energy Management
  6. Contributions
  7. Method
  8. Reinforcement Learning with Decision Transformers
  9. Knowledge Distillation for Model Compression
  10. Evaluation
  11. Problem Formulation -- Environment
  12. Data and Hardware Setup
  13. Used Dataset
  14. Offline Dataset Generation
  15. Used Hardware
  16. ...and 11 more sections

Figures (9)

  • Figure 1: Markov Decision Process for Reinforcement Learning in a Battery Scheduling Environment.
  • Figure 2: Decision Transformer in a Battery Scheduling Environment (adapted from chen2021).
  • Figure 3: Response-Based Knowledge Distillation for Decision Transformer in a Battery Scheduling Environment.
  • Figure 4: Aggregated Daily Mean Load for Building 13 with a Typical Residential Load Pattern.
  • Figure 5: Mean Cost Across All 20 Buildings for Different Approaches. DT Denotes the Medium Decision Transformer, while KD Refers to Knowledge Distillation From Medium Teacher to Small Student.
  • ...and 4 more figures