Personalized Federated Deep Reinforcement Learning for Heterogeneous Edge Content Caching Networks

TL;DR

This work tackles proactive caching in heterogeneous edge networks by introducing a scalable decision framework that jointly combats the curse of dimensionality and environment diversity. It proposes Multi-head DQN (MH-DQN) to reduce the discrete action space from $2^C$ to $2C$ for per-content decisions, and a layer-wise personalized federated learning (PF-DRL-Ca) architecture that aggregates only base layers while preserving personalized heads for each MEC server. The approach yields faster convergence, higher cache hit ratios, and lower replacement costs in single-server tests, and demonstrably superior performance in heterogeneous multi-server settings due to learned global representations plus local adaptation. The results underscore the practical impact for real-world edge caching, enabling scalable, privacy-preserving collaboration across edge nodes with tailored caching policies.

Abstract

Proactive caching is essential for minimizing latency and improving Quality of Experience (QoE) in multi-server edge networks. Federated Deep Reinforcement Learning (FDRL) is a promising approach for developing cache policies tailored to dynamic content requests. However, FDRL faces challenges such as an expanding caching action space due to increased content numbers and difficulty in adapting global information to heterogeneous edge environments. In this paper, we propose a Personalized Federated Deep Reinforcement Learning framework for Caching, called PF-DRL-Ca, with the aim to maximize system utility while satisfying caching capability constraints. To manage the expanding action space, we employ a new DRL algorithm, Multi-head Deep Q-Network (MH-DQN), which reshapes the action output layers of DQN into a multi-head structure where each head generates a sub-dimensional action. We next integrate the proposed MH-DQN into a personalized federated training framework, employing a layer-wise approach for training to derive a personalized model that can adapt to heterogeneous environments while exploiting the global information to accelerate learning convergence. Our extensive experimental results demonstrate the superiority of MH-DQN over traditional DRL algorithms on a single server, as well as the advantages of the personal federated training architecture compared to other frameworks.

Figures (8)

• Figure 1: System model.
• Figure 2: Overview of proposed PF-DRL-Ca framework.
• Figure 3: Performance comparison of modified DQN and PPO algorithm.
• Figure 4: Performance comparison with different numbers of personalized layers.
• Figure 5: Convergence comparison of three DRL-based algorithms.