Combinatorial Client-Master Multiagent Deep Reinforcement Learning for Task Offloading in Mobile Edge Computing

TL;DR

CCM_MADRL_MEC is the first MADRL approach in task offloading to consider server storage capacity in addition to the constraints of the UDs, and has shown superior convergence over existing benchmark and heuristic algorithms.

Abstract

Recently, there has been an explosion of mobile applications that perform computationally intensive tasks such as video streaming, data mining, virtual reality, augmented reality, image processing, video processing, face recognition, and online gaming. However, user devices (UDs), such as tablets and smartphones, have a limited ability to perform the computation needs of the tasks. Mobile edge computing (MEC) has emerged as a promising technology to meet the increasing computing demands of UDs. Task offloading in MEC is a strategy that meets the demands of UDs by distributing tasks between UDs and MEC servers. Deep reinforcement learning (DRL) is gaining attention in task-offloading problems because it can adapt to dynamic changes and minimize online computational complexity. However, the various types of continuous and discrete resource constraints on UDs and MEC servers pose challenges to the design of an efficient DRL-based task-offloading strategy. Existing DRL-based task-offloading algorithms focus on the constraints of the UDs, assuming the availability of enough storage resources on the server. Moreover, existing multiagent DRL (MADRL)--based task-offloading algorithms are homogeneous agents and consider homogeneous constraints as a penalty in their reward function. We proposed a novel combinatorial client-master MADRL (CCM\_MADRL) algorithm for task offloading in MEC (CCM\_MADRL\_MEC) that enables UDs to decide their resource requirements and the server to make a combinatorial decision based on the requirements of the UDs. CCM\_MADRL\_MEC is the first MADRL in task offloading to consider server storage capacity in addition to the constraints in the UDs. By taking advantage of the combinatorial action selection, CCM\_MADRL\_MEC has shown superior convergence over existing MADDPG and heuristic algorithms.

Figures (5)

• Figure 1: Network model
• Figure 2: The interaction diagram of the agents and the MEC environment. Client agents output their actions $\{x_{n}, p_n, f_{n}\}$. Clients with $\{ x_{c,n} < 0.5$} start local processing; and the others propose their tasks to the master agent, who makes the combinatorial decision on which of the proposed tasks should be offloaded and which of them should be designated for local processing
• Figure 3: Comparison of the CCM_MADRL in the evaluation environment with the heuristic and MADDPG algorithms with a learning rate of 0.0001 and 0.001 for the clients and master respectively
• Figure 4: Comparison of the CCM_MADRL with the heuristic and MADDPG algorithms on the evaluation environment with learning rates of 0.0001 and 0.001 for the clients and master respectively, and $b_{max}$ = $b_{min}$+$1J$
• Figure 5: Comparison of the CCM_MADRL with the heuristic and MADDPG algorithms on the evaluation environment with learning rates of 0.0001 and 0.001 for the clients and master respectively, $b_{max}$ = $b_{min}$+$1J$, and $\lambda_2$ = 5