Efficient Implementation of Reinforcement Learning over Homomorphic Encryption

TL;DR

This work tackles privacy-preserving reinforcement learning for cloud-based control policy synthesis by leveraging fully homomorphic encryption (FHE). It analyzes standard RL methods and their incompatibility with ciphertext-level comparisons, and then highlights Relative-Entropy-regularized RL (RE-RL) as a min-free framework that enables ciphertext-only implementations across model-based, simulator-driven, and data-driven paradigms. The paper develops three RE-RL realizations—linearly solvable value iteration, path integral control, and Z-learning—and demonstrates their encryptable updates, along with a numerical study in encrypted Z-learning on a CKKS-based grid world to illustrate convergence and approximation behavior. The results suggest that RE-RL can enable secure, efficient cloud-based RL for control tasks, reducing reliance on plaintext comparisons and opening practical avenues for privacy-preserving robotic and control applications.

Abstract

We investigate encrypted control policy synthesis over the cloud. While encrypted control implementations have been studied previously, we focus on the less explored paradigm of privacy-preserving control synthesis, which can involve heavier computations ideal for cloud outsourcing. We classify control policy synthesis into model-based, simulator-driven, and data-driven approaches and examine their implementation over fully homomorphic encryption (FHE) for privacy enhancements. A key challenge arises from comparison operations (min or max) in standard reinforcement learning algorithms, which are difficult to execute over encrypted data. This observation motivates our focus on Relative-Entropy-regularized reinforcement learning (RL) problems, which simplifies encrypted evaluation of synthesis algorithms due to their comparison-free structures. We demonstrate how linearly solvable value iteration, path integral control, and Z-learning can be readily implemented over FHE. We conduct a case study of our approach through numerical simulations of encrypted Z-learning in a grid world environment using the CKKS encryption scheme, showing convergence with acceptable approximation error. Our work suggests the potential for secure and efficient cloud-based reinforcement learning.

Figures (3)

• Figure 1: Encrypted policy implementation (left) and encrypted policy synthesis (right).
• Figure 2: Grid World Value Progression (Z-Learning).
• Figure 3: Approximation Error Normalized.