Go-Explore for Residential Energy Management

TL;DR

The paper tackles the challenge of reinforcement learning in residential energy management where deceptive and sparse rewards hinder exploration. It applies Go-Explore, a planning-plus-RL framework that memorizes promising states and revisits them to overcome detachment and derailment, to a cost-saving energy management task. The approach yields up to 19.84% relative improvement over a DQN baseline, with Go-Explore showing robust performance across phase 1 exploration and phase 2 robustification, and uses real-world datasets for price, renewable generation, and background loads. This work demonstrates the viability of integrating planning-based exploration with RL in real-life energy systems and points to future work in policy-based Go-Explore and multi-agent extensions for broader applicability.

Abstract

Reinforcement learning is commonly applied in residential energy management, particularly for optimizing energy costs. However, RL agents often face challenges when dealing with deceptive and sparse rewards in the energy control domain, especially with stochastic rewards. In such situations, thorough exploration becomes crucial for learning an optimal policy. Unfortunately, the exploration mechanism can be misled by deceptive reward signals, making thorough exploration difficult. Go-Explore is a family of algorithms which combines planning methods and reinforcement learning methods to achieve efficient exploration. We use the Go-Explore algorithm to solve the cost-saving task in residential energy management problems and achieve an improvement of up to 19.84\% compared to the well-known reinforcement learning algorithms.