Conditional Diffusion Model with OOD Mitigation as High-Dimensional Offline Resource Allocation Planner in Clustered Ad Hoc Networks

TL;DR

This work tackles the challenge of high-dimensional resource allocation in clustered ad hoc networks, where the action space scales as $N^{M\times L}$ and online interaction costs are prohibitive. It introduces CDMP, a model-based offline planner that uses a conditional diffusion model to capture environmental dynamics and an inverse dynamics model to generate actions, supplemented by CDMP-pen, an uncertainty-aware penalty based on the smoothed distance to data to mitigate OOD distribution shifts. The authors provide theoretical guarantees linking the smoothed data-distance metric to a bound on model uncertainty via a Lipschitz constant, and demonstrate through OPNET-based experiments that CDMP outperforms model-free RL baselines and is competitive with state-of-the-art offline MBRL methods, with CDMP-pen offering enhanced robustness under dinamic interference. The proposed approach offers a sample-efficient, uncertainty-aware framework for high-dimensional offline planning in communication networks, with potential extensions to more scalable and distributed scheduling solutions.

Abstract

Due to network delays and scalability limitations, clustered ad hoc networks widely adopt Reinforcement Learning (RL) for on-demand resource allocation. Albeit its demonstrated agility, traditional Model-Free RL (MFRL) solutions struggle to tackle the huge action space, which generally explodes exponentially along with the number of resource allocation units, enduring low sampling efficiency and high interaction cost. In contrast to MFRL, Model-Based RL (MBRL) offers an alternative solution to boost sample efficiency and stabilize the training by explicitly leveraging a learned environment model. However, establishing an accurate dynamic model for complex and noisy environments necessitates a careful balance between model accuracy and computational complexity $\&$ stability. To address these issues, we propose a Conditional Diffusion Model Planner (CDMP) for high-dimensional offline resource allocation in clustered ad hoc networks. By leveraging the astonishing generative capability of Diffusion Models (DMs), our approach enables the accurate modeling of high-quality environmental dynamics while leveraging an inverse dynamics model to plan a superior policy. Beyond simply adopting DMs in offline RL, we further incorporate the CDMP algorithm with a theoretically guaranteed, uncertainty-aware penalty metric, which theoretically and empirically manifests itself in mitigating the Out-of-Distribution (OOD)-induced distribution shift issue underlying scarce training data. Extensive experiments also show that our model outperforms MFRL in average reward and Quality of Service (QoS) while demonstrating comparable performance to other MBRL algorithms.

Figures (11)

• Figure 1: Ad hoc network topology under MF-TDMA MAC protocol.
• Figure 2: Training and implementation of CDMP & CDMP-pen.
• Figure 3: Illustration of the perturbed two-dimensional swiss roll distribution.
• Figure 4: Comparison of CDMP with different methods in terms of average reward.
• Figure 5: Comparison of CDMP with different methods in terms of QoS. Specifically, static services represent a constant ratio of high- to low-speed nodes throughout the communication period. In contrast, dynamic services involve a changing ratio of high- to low-speed nodes over time.

Theorems & Definitions (2)

• Lemma 1
• Theorem 1