Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification

Nan Qiao, Sheng Yue

TL;DR

ForLER is presented, combining Q-ensemble aggregation on the server with actor rectification on devices to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy.

Abstract

In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be risky and costly, motivating offline federated RL (FRL), where local devices learn from fixed datasets. Despite its promise, offline FRL may break down under low-quality, heterogeneous data. Offline RL tends to get stuck in local optima, and in FRL, one device's suboptimal policy can degrade the aggregated model, i.e., policy pollution. We present FORLER, combining Q-ensemble aggregation on the server with actor rectification on devices. The server robustly merges device Q-functions to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy. Locally, actor rectification enriches policy gradients via a zeroth-order search for high-Q actions plus a bespoke regularizer that nudges the policy toward them. A $δ$-periodic strategy further reduces local computation. We theoretically provide safe policy improvement performance guarantees. Extensive experiments show FORLER consistently outperforms strong baselines under varying data quality and heterogeneity.

FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification

TL;DR

ForLER is presented, combining Q-ensemble aggregation on the server with actor rectification on devices to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy.

Abstract

In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be risky and costly, motivating offline federated RL (FRL), where local devices learn from fixed datasets. Despite its promise, offline FRL may break down under low-quality, heterogeneous data. Offline RL tends to get stuck in local optima, and in FRL, one device's suboptimal policy can degrade the aggregated model, i.e., policy pollution. We present FORLER, combining Q-ensemble aggregation on the server with actor rectification on devices. The server robustly merges device Q-functions to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy. Locally, actor rectification enriches policy gradients via a zeroth-order search for high-Q actions plus a bespoke regularizer that nudges the policy toward them. A -periodic strategy further reduces local computation. We theoretically provide safe policy improvement performance guarantees. Extensive experiments show FORLER consistently outperforms strong baselines under varying data quality and heterogeneity.
Paper Structure (17 sections, 1 theorem, 16 equations, 6 figures, 2 algorithms)

This paper contains 17 sections, 1 theorem, 16 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. FL and RL
  4. Deployment and dataset assumptions.
  5. Proposed Method
  6. The Motivating Example
  7. Local Updates with $\delta$-periodic Actor Rectification
  8. Local policy evaluation
  9. Local policy improvement with $\delta$-periodic Actor Rectification
  10. Server Aggregation with Q-Ensemble
  11. FORLER
  12. Theoretical Analysis
  13. Simulation
  14. Experimental Setup
  15. Experimental Evaluation
  16. ...and 2 more sections

Key Result

Theorem 1

Let $\pi_k^{*}$ be the policy obtained by optimizing Eq. eq:localpi. Define and let $d^{\pi_k}(\mathbf{s})$ denote the marginal discounted state-visitation distribution of policy $\pi_k$. Then, for $\alpha>0$ and $\eta\in(0,1)$,

Figures (6)

  • Figure 1: The architecture of FORLER. The system integrates a suite of devices coordinated by a server. Both the individual devices and the server are equipped with offline datasets and compute independently.
  • Figure 2: The performance of advanced algorithms, e.g. FEDORA. (Left) Overall performance difference between high-quality datasets and mixed (high-quality + additional low-quality) datasets. (Right) A visual representation of the Q-value vs. Action. The circle symbolizes the predicted action derived from the updating policy of FEDORA, while the pentacle represents the corresponding action predicted by the updating policy of our method.
  • Figure 3: The performance of FORLER against baselines.
  • Figure 4: Performance comparison of different methods on policy pollution problem.
  • Figure 5: Effect of devices, data, and hyperparameters.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1: Safe policy improvement
  • proof