Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data

Nan Qiao, Sheng Yue, Ju Ren, Yaoxue Zhang

TL;DR

Offline federated reinforcement learning faces severe performance drops when client data quality is heterogeneous. The authors propose FOVA, a framework that combines a vote-based local evaluation (VCQL) with advantage-weighted regression (AWR) for consistent local/global optimization, and proves policy-improvement guarantees under data heterogeneity. The method, paired with simple server aggregation, yields strong empirical gains on D4RL Gym locomotion and Maze2D benchmarks, especially in mixed-quality data settings. Overall, FOVA enables robust, privacy-preserving distributed offline RL with explicit guidance on tuning regularizers to balance cross-client aggregation and dataset mismatch.

Abstract

Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance of most existing offline FRL methods drops dramatically when provided with mixed-quality data, that is, the logging behaviors (offline data) are collected by policies with varying qualities across clients. To overcome this limitation, this paper introduces a new vote-based offline FRL framework, named FOVA. It exploits a \emph{vote mechanism} to identify high-return actions during local policy evaluation, alleviating the negative effect of low-quality behaviors from diverse local learning policies. Besides, building on advantage-weighted regression (AWR), we construct consistent local and global training objectives, significantly enhancing the efficiency and stability of FOVA. Further, we conduct an extensive theoretical analysis and rigorously show that the policy learned by FOVA enjoys strict policy improvement over the behavioral policy. Extensive experiments corroborate the significant performance gains of our proposed algorithm over existing baselines on widely used benchmarks.

FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data

TL;DR

Offline federated reinforcement learning faces severe performance drops when client data quality is heterogeneous. The authors propose FOVA, a framework that combines a vote-based local evaluation (VCQL) with advantage-weighted regression (AWR) for consistent local/global optimization, and proves policy-improvement guarantees under data heterogeneity. The method, paired with simple server aggregation, yields strong empirical gains on D4RL Gym locomotion and Maze2D benchmarks, especially in mixed-quality data settings. Overall, FOVA enables robust, privacy-preserving distributed offline RL with explicit guidance on tuning regularizers to balance cross-client aggregation and dataset mismatch.

Abstract

Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance of most existing offline FRL methods drops dramatically when provided with mixed-quality data, that is, the logging behaviors (offline data) are collected by policies with varying qualities across clients. To overcome this limitation, this paper introduces a new vote-based offline FRL framework, named FOVA. It exploits a \emph{vote mechanism} to identify high-return actions during local policy evaluation, alleviating the negative effect of low-quality behaviors from diverse local learning policies. Besides, building on advantage-weighted regression (AWR), we construct consistent local and global training objectives, significantly enhancing the efficiency and stability of FOVA. Further, we conduct an extensive theoretical analysis and rigorously show that the policy learned by FOVA enjoys strict policy improvement over the behavioral policy. Extensive experiments corroborate the significant performance gains of our proposed algorithm over existing baselines on widely used benchmarks.

Paper Structure

This paper contains 33 sections, 15 theorems, 125 equations, 10 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Federated Reinforcement Learning
  5. Offline Federated Reinforcement Learning
  6. System Model
  7. Proposed Method
  8. Local Policy Evaluation with Vote Mechanism
  9. Local Policy Improvement with AWR
  10. Server Aggregation
  11. Simulation
  12. Experiment Setup
  13. Experiment Results
  14. Overall Performance
  15. Consistent Optimization Objective
  16. ...and 18 more sections

Key Result

Lemma 1

The value of the policy under the Q-function from Eq. eq:vcql, $\hat{V}^{\pi^v_k}(\mathbf{s}) = \mathbb{E}_{{\pi^v_k}(\mathbf{a}|\mathbf{s})}[\hat{Q}^{{\pi^v_k}}(\mathbf{s}, \mathbf{a})]$, lower-bounds the true value of the policy obtained via exact policy evaluation, $V^{\pi^v_k}(\mathbf{s}) = \mat where $\Gamma_k = \left(I - \gamma P^{{\pi^v_k}} \right)$ is non-negative entries and $C_{v}=\frac{

Figures (10)

  • Figure 1: Results of CQL-FL park2022federated, FEDORA rengarajan2023federated, DRPO yue2024federated and the proposed method on Gym locomotion tasks. Left: Performance comparison between mixed-quality and uniform-quality (commonly used as evaluated benchmarks in the existing literature) cases. Right: Performance comparison between the average client-side (local policies) return and server-side (global policy) return.
  • Figure 2: Illustration of offline FRL training procedure, demonstrating how clients collaborate training based on our FOVA method.
  • Figure 3: Gym locomotion and Maze2D tasks.
  • Figure 4: Performance comparison between server-side policy and client-side policy.
  • Figure 5: Performance of FOVA against baselines.
  • ...and 5 more figures

Theorems & Definitions (38)

  • Lemma 1
  • proof
  • Remark 1
  • Lemma 2
  • proof
  • Theorem 1
  • proof
  • Remark 2
  • Theorem 2
  • proof
  • ...and 28 more