Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic and Distributed Routing in IoT Networks based on Multi-Objective Q-Learning

Shubham Vaishnav, Praveen Kumar Donta, Sindri Magnússon

TL;DR

The paper tackles dynamic, multi-objective routing in energy-constrained IoT networks by introducing DPQ-Learning, a fully distributed framework that learns multiple per-preference Q-tables in parallel. It adds a Greedy Interpolation Policy to enable near-optimal action selection for unseen preference weights without retraining or central coordination, backed by a Lipschitz-continuity based theoretical guarantee. The approach is implemented in a distributed manner across nodes, with convergence guarantees for the per-grid Q-tables and an interpolation bound that depends on grid resolution. Empirical results show substantial improvements in energy efficiency, packet delivery, and cumulative rewards over six baselines, highlighting real-time adaptability and scalability in dynamic IoT environments.

Abstract

IoT networks often face conflicting routing goals such as maximizing packet delivery, minimizing delay, and conserving limited battery energy. These priorities can also change dynamically: for example, an emergency alert requires high reliability, while routine monitoring prioritizes energy efficiency to prolong network lifetime. Existing works, including many deep reinforcement learning approaches, are typically centralized and assume static objectives, making them slow to adapt when preferences shift. We propose a dynamic and fully distributed multi-objective Q-learning routing algorithm that learns multiple per-preference Q-tables in parallel and introduces a novel greedy interpolation policy to act near-optimally for unseen preferences without retraining or central coordination. A theoretical analysis further shows that the optimal value function is Lipschitz-continuous in the preference parameter, ensuring that the proposed greedy interpolation policy yields provably near-optimal behavior. Simulations show that our approach adapts in real time to shifting priorities and achieves up to 80-90\% lower energy consumption and more than 2-5x higher cumulative rewards and packet delivery compared to six baseline protocols. These results demonstrate significant gains in adaptability, delivery, and efficiency for dynamic IoT environments.

Dynamic and Distributed Routing in IoT Networks based on Multi-Objective Q-Learning

TL;DR

The paper tackles dynamic, multi-objective routing in energy-constrained IoT networks by introducing DPQ-Learning, a fully distributed framework that learns multiple per-preference Q-tables in parallel. It adds a Greedy Interpolation Policy to enable near-optimal action selection for unseen preference weights without retraining or central coordination, backed by a Lipschitz-continuity based theoretical guarantee. The approach is implemented in a distributed manner across nodes, with convergence guarantees for the per-grid Q-tables and an interpolation bound that depends on grid resolution. Empirical results show substantial improvements in energy efficiency, packet delivery, and cumulative rewards over six baselines, highlighting real-time adaptability and scalability in dynamic IoT environments.

Abstract

IoT networks often face conflicting routing goals such as maximizing packet delivery, minimizing delay, and conserving limited battery energy. These priorities can also change dynamically: for example, an emergency alert requires high reliability, while routine monitoring prioritizes energy efficiency to prolong network lifetime. Existing works, including many deep reinforcement learning approaches, are typically centralized and assume static objectives, making them slow to adapt when preferences shift. We propose a dynamic and fully distributed multi-objective Q-learning routing algorithm that learns multiple per-preference Q-tables in parallel and introduces a novel greedy interpolation policy to act near-optimally for unseen preferences without retraining or central coordination. A theoretical analysis further shows that the optimal value function is Lipschitz-continuous in the preference parameter, ensuring that the proposed greedy interpolation policy yields provably near-optimal behavior. Simulations show that our approach adapts in real time to shifting priorities and achieves up to 80-90\% lower energy consumption and more than 2-5x higher cumulative rewards and packet delivery compared to six baseline protocols. These results demonstrate significant gains in adaptability, delivery, and efficiency for dynamic IoT environments.
Paper Structure (25 sections, 2 theorems, 37 equations, 8 figures, 1 table, 2 algorithms)

This paper contains 25 sections, 2 theorems, 37 equations, 8 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Contributions and Novelty
  4. Related Works
  5. System Model and Problem Formulation
  6. Network Model and Routing Algorithms
  7. Routing Dynamic Preference
  8. Learning Based Routing with Dynamic Preferences
  9. Q-Learning
  10. Dynamic Preference Q-Learning
  11. Exploration Strategies
  12. Distributed DPQ-Learning
  13. Performance Guarantees for the GIP
  14. Proof of Theorem \ref{['Thm:1']}
  15. Experimental Results
  16. ...and 10 more sections

Key Result

Theorem 1

Under Assumption assump:pergrid-accuracy, for any $\beta\in [0,1]$ the interpolation policy in Eq. eq:Qint where $\underline{\beta} \triangleq \max \{ b \in \mathcal{B} | b\leq \beta \}$ and $\overline{\beta}\triangleq \min \{ b \in \mathcal{B} | b\geq \beta \}$.

Figures (8)

  • Figure 1: Depiction of Proposed Distributed DPQ-Learning Routing
  • Figure 2: Variation of Epsilon and Preferences during the experimental simulations.
  • Figure 3: Overall reward in Sequential exploration-exploitation scheme.
  • Figure 4: Energy Consumption in the Sequential exploration-exploitation scheme.
  • Figure 5: PDR in the Sequential exploration-exploitation scheme.
  • ...and 3 more figures

Theorems & Definitions (4)

  • Remark 1
  • Theorem 1
  • Lemma 1
  • proof