Deep Q-Learning Assisted Bandwidth Reservation for Multi-Operator Time-Sensitive Vehicular Networking

TL;DR

The paper tackles bandwidth reservation for time-sensitive vehicular networking across multiple MNOs under dynamic pricing. It models the problem as a Markov Decision Process and solves it with a Double Deep Q-Learning (DDQL) agent that learns when and how to update reservations to minimize total cost while addressing exact, underbooking, overbooking, and mixed scenarios. Key contributions include the formal MDP formulation, a reward design incorporating cancellation and booking costs, and empirical evidence showing up to a 40% cost reduction over baselines using historical pricing data. The approach supports edge/cloud deployment and enhances reliable resource provisioning for safety-critical V2X applications in dynamic, multi-operator environments.

Abstract

Very few available individual bandwidth reservation schemes provide efficient and cost-effective bandwidth reservation that is required for safety-critical and time-sensitive vehicular networked applications. These schemes allow vehicles to make reservation requests for the required resources. Accordingly, a Mobile Network Operator (MNO) can allocate and guarantee bandwidth resources based on these requests. However, due to uncertainty in future reservation time and bandwidth costs, the design of an optimized reservation strategy is challenging. In this article, we propose a novel multi-objective bandwidth reservation update approach with an optimal strategy based on Double Deep Q-Network (DDQN). The key design objectives are to minimize the reservation cost with multiple MNOs and to ensure reliable resource provisioning in uncertain situations by solving scenarios such as underbooked and overbooked reservations along the driving path. The enhancements and advantages of our proposed strategy have been demonstrated through extensive experimental results when compared to other methods like greedy update or other deep reinforcement learning approaches. Our strategy demonstrates a 40% reduction in bandwidth costs across all investigated scenarios and simultaneously resolves uncertain situations in a cost-effective manner.

Figures (9)

• Figure 1: System model and an example of exactbooking, underbooking and overbooking scenarios.
• Figure 2: An illustration of the proposed double Q-learning algorithm for bandwidth reservation update strategy.
• Figure 3: Learning curves showing the convergence of the different DQN methods, demonstrating that the suggested Double DQN algorithm yields the highest average episode rewards.
• Figure 4: Cumulative episode rewards are significantly lower with suggested algorithms compared to baseline approaches across various scenarios: Exact booking (a), underbooking (b), overbooking (c), and mixed booking (d). The figure also illustrates the cumulative reservation costs for all scenarios.
• Figure 5: Average costs per episode for all scenarios and policies. According to the figure, the suggested Double DQN policy has the lowest cost in all scenarios.