Federated Q-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Cost

TL;DR

The paper introduces FedQ-Advantage, a model-free federated Q-learning algorithm for tabular episodic MDPs that achieves almost-optimal regret up to logarithmic factors and logarithmic communication costs. It leverages reference-advantage decomposition to reduce variance and employs event-triggered, heterogeneous synchronization combined with policy updates to balance exploration and communication. The method achieves near-linear speedup in the number of agents and outperforms prior federated Q-learning approaches in both regret and communication efficiency. The key innovations include stage-wise analysis for non-martingales and a flexible synchronization mechanism that adapts communication rounds to exploration dynamics, enabling scalable FRL with strong theoretical guarantees and practical efficiency.

Abstract

In this paper, we consider model-free federated reinforcement learning for tabular episodic Markov decision processes. Under the coordination of a central server, multiple agents collaboratively explore the environment and learn an optimal policy without sharing their raw data. Despite recent advances in federated Q-learning algorithms achieving near-linear regret speedup with low communication cost, existing algorithms only attain suboptimal regrets compared to the information bound. We propose a novel model-free federated Q-learning algorithm, termed FedQ-Advantage. Our algorithm leverages reference-advantage decomposition for variance reduction and operates under two distinct mechanisms: synchronization between the agents and the server, and policy update, both triggered by events. We prove that our algorithm not only requires a lower logarithmic communication cost but also achieves an almost optimal regret, reaching the information bound up to a logarithmic factor and near-linear regret speedup compared to its single-agent counterpart when the time horizon is sufficiently large.

Figures (4)

• Figure 1: The relationship between rounds and stages for different triples $(s^1,a^1,h^1)$ and $(s^2,a^2,h^2)$. Each square represents a round, and the number inside indicates the round index. A stage is composed of consecutive rounds. Communication occurs at the end of each round and the estimated $Q$-function is updated at the end of each stage. We can find from the figure that a round may belong to different stages for different triples. For example, the round $k_{11}$ is in stage $1$ of $(s^1,a^1,h^1)$, while in stage $2$ of $(s^2,a^2,h^2)$. Here, $k_{it} = k_{h^i}^{t+1}(s^i,a^i) - 1$ represents the index of the last round in stage $t$ for $(s^i,a^i,h^i)$, $t\in \{1,2,\cdots,T_i\}$ and $i\in \{1,2\}$. $T_1$ and $T_2$ are the total number of stages for $(s^1,a^1,h^1)$ and $(s^2,a^2,h^2)$ respectively.
• Figure 2: Numerical comparison of regrets and communication costs.
• Figure 3: Multi-agent speedup
• Figure 4: Additional Experiments. The upper panels (a) and (b) replicate the numerical setting in zheng2023federated where $S=3,A=2,H=5$. The bottom panels (c) and (d) perform experiments with $H=20,S=20,A=5$.

Theorems & Definitions (27)

• Theorem 4.1: Regret of FedQ-Advantage
• Theorem 4.2: Communication rounds of FedQ-Advantage
• Lemma D.1
• proof : Proof of Lemma \ref{['algorithm relationship']}
• Lemma D.2
• proof : Proof of Lemma \ref{['n-N']}
• Lemma D.3
• Lemma D.4
• Lemma D.5
• proof : Proof of Lemma \ref{['concentrations']}