Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybrid Cognitive IoT with Cooperative Caching and SWIPT-EH: A Hierarchical Reinforcement Learning Framework

Nadia Abdolkhani, Walaa Hamouda

TL;DR

This work introduces a three-level hierarchical reinforcement learning framework (H-SAC) for CIoT networks that jointly optimizes energy harvesting, hybrid overlay-underlay spectrum access, power control, and cooperative caching under SWIPT-EH constraints. By decomposing decisions into high-level time-switching, mid-level access coordination, and low-level power/caching actions, the approach handles mixed discrete-continuous actions with improved scalability and learning stability. Empirical results show that H-SAC outperforms flat DRL baselines in average sum rate, cache hit rate, cooperation with PU, and delay reduction, while maintaining energy efficiency under channel fading and uncertain PU activity. The work highlights the critical role of adaptive EH management and caching-enabled spectrum cooperation for practical CIoT deployments, and it points to future extensions including non-linear EH models and multi-agent coordination.

Abstract

This paper proposes a hierarchical deep reinforcement learning (DRL) framework based on the soft actor-critic (SAC) algorithm for hybrid underlay-overlay cognitive Internet of Things (CIoT) networks with simultaneous wireless information and power transfer (SWIPT)-energy harvesting (EH) and cooperative caching. Unlike prior hierarchical DRL approaches that focus primarily on spectrum access or power control, our work jointly optimizes EH, hybrid access coordination, power allocation, and caching in a unified framework. The joint optimization problem is formulated as a weighted-sum multi-objective task, designed to maximize throughput and cache hit ratio while simultaneously minimizing transmission delay. In the proposed model, CIoT agents jointly optimize EH and data transmission using a learnable time switching (TS) factor. They also coordinate spectrum access under hybrid overlay-underlay paradigms and make power control and cache placement decisions while considering energy, interference, and storage constraints. Specifically, in this work, cooperative caching is used to enable overlay access, while power control is used for underlay access. A novel three-level hierarchical SAC (H-SAC) agent decomposes the mixed discrete-continuous action space into modular subproblems, improving scalability and convergence over flat DRL methods. The high-level policy adjusts the TS factor, the mid-level policy manages spectrum access coordination and cache sharing, and the low-level policy decides transmit power and caching actions for both the CIoT agent and PU content. Simulation results show that the proposed hierarchical SAC approach significantly outperforms benchmark and greedy strategies. It achieves better performance in terms of average sum rate, delay, cache hit ratio, and energy efficiency, even under channel fading and uncertain conditions.

Hybrid Cognitive IoT with Cooperative Caching and SWIPT-EH: A Hierarchical Reinforcement Learning Framework

TL;DR

This work introduces a three-level hierarchical reinforcement learning framework (H-SAC) for CIoT networks that jointly optimizes energy harvesting, hybrid overlay-underlay spectrum access, power control, and cooperative caching under SWIPT-EH constraints. By decomposing decisions into high-level time-switching, mid-level access coordination, and low-level power/caching actions, the approach handles mixed discrete-continuous actions with improved scalability and learning stability. Empirical results show that H-SAC outperforms flat DRL baselines in average sum rate, cache hit rate, cooperation with PU, and delay reduction, while maintaining energy efficiency under channel fading and uncertain PU activity. The work highlights the critical role of adaptive EH management and caching-enabled spectrum cooperation for practical CIoT deployments, and it points to future extensions including non-linear EH models and multi-agent coordination.

Abstract

This paper proposes a hierarchical deep reinforcement learning (DRL) framework based on the soft actor-critic (SAC) algorithm for hybrid underlay-overlay cognitive Internet of Things (CIoT) networks with simultaneous wireless information and power transfer (SWIPT)-energy harvesting (EH) and cooperative caching. Unlike prior hierarchical DRL approaches that focus primarily on spectrum access or power control, our work jointly optimizes EH, hybrid access coordination, power allocation, and caching in a unified framework. The joint optimization problem is formulated as a weighted-sum multi-objective task, designed to maximize throughput and cache hit ratio while simultaneously minimizing transmission delay. In the proposed model, CIoT agents jointly optimize EH and data transmission using a learnable time switching (TS) factor. They also coordinate spectrum access under hybrid overlay-underlay paradigms and make power control and cache placement decisions while considering energy, interference, and storage constraints. Specifically, in this work, cooperative caching is used to enable overlay access, while power control is used for underlay access. A novel three-level hierarchical SAC (H-SAC) agent decomposes the mixed discrete-continuous action space into modular subproblems, improving scalability and convergence over flat DRL methods. The high-level policy adjusts the TS factor, the mid-level policy manages spectrum access coordination and cache sharing, and the low-level policy decides transmit power and caching actions for both the CIoT agent and PU content. Simulation results show that the proposed hierarchical SAC approach significantly outperforms benchmark and greedy strategies. It achieves better performance in terms of average sum rate, delay, cache hit ratio, and energy efficiency, even under channel fading and uncertain conditions.

Paper Structure

This paper contains 15 sections, 28 equations, 12 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motivation and Contributions
  4. System Model
  5. Cognitive IoT Network
  6. Transmission Model
  7. Caching Model
  8. Energy Harvesting Model
  9. Problem Formulation: A Weighted-Sum Multi-Objective Problem
  10. Proposed Method
  11. The Model-Free Markov Decision Process
  12. Proposed Hierarchical Soft Actor-Critic Strategy
  13. Complexity Analysis
  14. Simulation Model and Results
  15. Conclusions

Figures (12)

  • Figure 1: System Model
  • Figure 2: Proposed H-SAC Strategy
  • Figure 3: Benchmarking the achievable reward of our proposed H-SAC strategy in comparison to the existing strategies.
  • Figure 4: Benchmarking the ASR performance of our proposed H-SAC strategy in comparison to other existing strategies.
  • Figure 5: Benchmarking the CIoT hit rate of our proposed H-SAC strategy in comparison to other existing strategies.
  • ...and 7 more figures