Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HRL-TSCH: A Hierarchical Reinforcement Learning-based TSCH Scheduler for IIoT

F. Fernando Jurado-Lasso, Charalampos Orfanidis, J. F. Jurado, Xenofon Fafoutis

TL;DR

The paper tackles the challenge of adaptable TSCH scheduling in IIoT networks, where application requirements demand balancing throughput, delay, and power. It introduces HRL-TSCH, a hierarchical reinforcement learning framework with a higher-level policy for link management and lower-level policies for slot/channel assignment, guided by a multi-objective cost function and trained via Deep Q-Networks. A TSCH lookup component translates RL decisions into concrete schedule cells, and a comprehensive evaluation in Contiki-NG/Cooja demonstrates superior trade-offs over centralized and decentralized baselines (Orchestra, MSF, QL-TSCH) across power, latency, and throughput, validated through Pareto-front analyses. The work highlights HRL-TSCH as a scalable, adaptable solution for dynamic IIoT environments, with potential extensions to slotframe size and contention-based scheduling to further enhance robustness and performance.

Abstract

The Industrial Internet of Things (IIoT) demands adaptable Networked Embedded Systems (NES) for optimal performance. Combined with recent advances in Artificial Intelligence (AI), tailored solutions can be developed to meet specific application requirements. This study introduces HRL-TSCH, an approach rooted in Hierarchical Reinforcement Learning (HRL), to devise Time Slotted Channel Hopping (TSCH) schedules provisioning IIoT demand. HRL-TSCH employs dual policies: one at a higher level for TSCH schedule link management, and another at a lower level for timeslot and channel assignments. The proposed RL agents address a multi-objective problem, optimizing throughput, power efficiency, and network delay based on predefined application requirements. Simulation experiments demonstrate HRL-TSCH superiority over existing state-of-art approaches, effectively achieving an optimal balance between throughput, power consumption, and delay, thereby enhancing IIoT network performance.

HRL-TSCH: A Hierarchical Reinforcement Learning-based TSCH Scheduler for IIoT

TL;DR

The paper tackles the challenge of adaptable TSCH scheduling in IIoT networks, where application requirements demand balancing throughput, delay, and power. It introduces HRL-TSCH, a hierarchical reinforcement learning framework with a higher-level policy for link management and lower-level policies for slot/channel assignment, guided by a multi-objective cost function and trained via Deep Q-Networks. A TSCH lookup component translates RL decisions into concrete schedule cells, and a comprehensive evaluation in Contiki-NG/Cooja demonstrates superior trade-offs over centralized and decentralized baselines (Orchestra, MSF, QL-TSCH) across power, latency, and throughput, validated through Pareto-front analyses. The work highlights HRL-TSCH as a scalable, adaptable solution for dynamic IIoT environments, with potential extensions to slotframe size and contention-based scheduling to further enhance robustness and performance.

Abstract

The Industrial Internet of Things (IIoT) demands adaptable Networked Embedded Systems (NES) for optimal performance. Combined with recent advances in Artificial Intelligence (AI), tailored solutions can be developed to meet specific application requirements. This study introduces HRL-TSCH, an approach rooted in Hierarchical Reinforcement Learning (HRL), to devise Time Slotted Channel Hopping (TSCH) schedules provisioning IIoT demand. HRL-TSCH employs dual policies: one at a higher level for TSCH schedule link management, and another at a lower level for timeslot and channel assignments. The proposed RL agents address a multi-objective problem, optimizing throughput, power efficiency, and network delay based on predefined application requirements. Simulation experiments demonstrate HRL-TSCH superiority over existing state-of-art approaches, effectively achieving an optimal balance between throughput, power consumption, and delay, thereby enhancing IIoT network performance.
Paper Structure (32 sections, 9 equations, 12 figures, 4 tables, 2 algorithms)

This paper contains 32 sections, 9 equations, 12 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Time Slotted Channel Hopping (TSCH)
  4. Reinforcement Learning (RL)
  5. Deep Q-Network (DQN)
  6. Hierarchical Reinforcement Learning (HRL)
  7. tsch and rl
  8. Related Work
  9. Centralized Schedulers
  10. Decentralized Schedulers
  11. Reinforcement Learning
  12. Reinforcement Learning Overview
  13. Hierarchical Reinforcement Learning Architecture
  14. Cost Function
  15. Higher-Level Policy
  16. ...and 17 more sections

Figures (12)

  • Figure 1: Reinforcement learning architecture.
  • Figure 2: Network topology and wireless link visualization. (a) Network topology with the sink node and sensor nodes. (b) Wireless link visualization showing the transmission range and interference range.
  • Figure 3: Pareto front that shows the trade-offs between power consumption, delay, and throughput. Fig. \ref{['fig:pareto_front_power_consumption']} shows the trade-offs between power consumption and delay. Fig. \ref{['fig:pareto_front_latency']} shows the trade-offs between throughput and delay. Fig. \ref{['fig:pareto_front_throughput']} shows the trade-offs between throughput and power consumption.
  • Figure 4: Common legend for Fig. \ref{['fig:performance']}, Fig. \ref{['fig:plr']}, Fig. \ref{['fig:node_power']}, Fig. \ref{['fig:node_throughput']}, Fig. \ref{['fig:node_jitter']} and Fig. \ref{['fig:node_packet_loss']}.
  • Figure 5: Performance comparison of hrltsch and baselines for network power consumption, delay, and throughput (see Fig. \ref{['fig:legend']} for the legend). (a) shows the average power consumption of the network. (b) shows the average delay of the network. (c) shows the average throughput of the network.
  • ...and 7 more figures