Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Offline Policy Optimization with Dual Regularization

Sheng Yue, Zerui Qin, Xingyuan Hua, Yongheng Deng, Ju Ren

TL;DR

This work tackles offline federated reinforcement learning by addressing two-tier distribution shifts that arise when learning from decentralized, static data. It introduces DRPO, a doubly regularized offline FRL algorithm that constrains local updates with respect to both the local behavioral policy and the globally aggregated policy, while using a conservative, off-policy Q-estimation mechanism. The authors prove strict policy-improvement guarantees and provide bounds that relate empirical and true MDP performance, demonstrating that proper balancing of the two regularizers yields reliable improvement. Empirically, DRPO outperforms baselines on standard offline RL benchmarks, with fast convergence and favorable communication efficiency, and ablations confirm the necessity of both regularizers. These results suggest DRPO as a practical and theoretically grounded approach for data-private, safety-sensitive distributed RL applications.

Abstract

Federated Reinforcement Learning (FRL) has been deemed as a promising solution for intelligent decision-making in the era of Artificial Internet of Things. However, existing FRL approaches often entail repeated interactions with the environment during local updating, which can be prohibitively expensive or even infeasible in many real-world domains. To overcome this challenge, this paper proposes a novel offline federated policy optimization algorithm, named $\texttt{DRPO}$, which enables distributed agents to collaboratively learn a decision policy only from private and static data without further environmental interactions. $\texttt{DRPO}$ leverages dual regularization, incorporating both the local behavioral policy and the global aggregated policy, to judiciously cope with the intrinsic two-tier distributional shifts in offline FRL. Theoretical analysis characterizes the impact of the dual regularization on performance, demonstrating that by achieving the right balance thereof, $\texttt{DRPO}$ can effectively counteract distributional shifts and ensure strict policy improvement in each federative learning round. Extensive experiments validate the significant performance gains of $\texttt{DRPO}$ over baseline methods.

Federated Offline Policy Optimization with Dual Regularization

TL;DR

This work tackles offline federated reinforcement learning by addressing two-tier distribution shifts that arise when learning from decentralized, static data. It introduces DRPO, a doubly regularized offline FRL algorithm that constrains local updates with respect to both the local behavioral policy and the globally aggregated policy, while using a conservative, off-policy Q-estimation mechanism. The authors prove strict policy-improvement guarantees and provide bounds that relate empirical and true MDP performance, demonstrating that proper balancing of the two regularizers yields reliable improvement. Empirically, DRPO outperforms baselines on standard offline RL benchmarks, with fast convergence and favorable communication efficiency, and ablations confirm the necessity of both regularizers. These results suggest DRPO as a practical and theoretically grounded approach for data-private, safety-sensitive distributed RL applications.

Abstract

Federated Reinforcement Learning (FRL) has been deemed as a promising solution for intelligent decision-making in the era of Artificial Internet of Things. However, existing FRL approaches often entail repeated interactions with the environment during local updating, which can be prohibitively expensive or even infeasible in many real-world domains. To overcome this challenge, this paper proposes a novel offline federated policy optimization algorithm, named , which enables distributed agents to collaboratively learn a decision policy only from private and static data without further environmental interactions. leverages dual regularization, incorporating both the local behavioral policy and the global aggregated policy, to judiciously cope with the intrinsic two-tier distributional shifts in offline FRL. Theoretical analysis characterizes the impact of the dual regularization on performance, demonstrating that by achieving the right balance thereof, can effectively counteract distributional shifts and ensure strict policy improvement in each federative learning round. Extensive experiments validate the significant performance gains of over baseline methods.
Paper Structure (24 sections, 4 theorems, 46 equations, 9 figures, 2 tables, 1 algorithm)

This paper contains 24 sections, 4 theorems, 46 equations, 9 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background and Challenge
  4. Markov Decision Processes and Reinforcement Learning
  5. Offline Federated Reinforcement Learning
  6. Challenges
  7. Federated Offline Policy Optimization
  8. Theoretical Analysis
  9. Notations and Assumptions
  10. Main Results: Strict Policy Improvement
  11. Improvement over aggregated policy
  12. Improvement over behavioral policy
  13. Experiments
  14. Experimental Setup
  15. Datasets
  16. ...and 9 more sections

Key Result

Theorem 1

Suppose that asmp:concentration, $(1-\delta_\pi)\lambda_2<\lambda_1<\lambda_2$, and $\delta_\pi>0$ (defined in Eq. (eq:delta_pi)) holds. Then, the optimal solution $\pi^*$ of Problem (eq:local_updating_pi) satisfies with probability great than $1-2\delta$ and a sufficient large $\lambda_2$.

Figures (9)

  • Figure 1: Online FRL vs Offline FRL.
  • Figure 2: Performance comparison of centralized CQL and naive federated CQL on two challenging MuJoCo tasks. Federated CQL follows FedAvg by using the CQL objective as the local objective. In Federated CQL, the number of agents is set as 10, and each agent has 5 medium-replay trajectories sampled from D4RL fu2020d4rl. The centralized CQL uses all 50 trajectories.
  • Figure 3: An illustration of the two-tier distributional shifts. $\mathcal{D}_i$ and $\mathcal{D}_j$ represent the state-action (data) distributions in local datasets. $\rho^\pi_i$ and $\rho^\pi_j$ denote the state-action distributions of local policies $\pi_i$ and $\pi_j$ in the environment.
  • Figure 4: Benchmark environments.
  • Figure 5: Impact of the number of agents on performance.
  • ...and 4 more figures

Theorems & Definitions (7)

  • Theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof