Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resource Allocation for Stable LLM Training in Mobile Edge Computing

Chang Liu, Jun Zhao

TL;DR

A collaborative training framework that integrates mobile users with edge servers to optimize resource allocation, thereby enhancing both performance and efficiency and addressing the common issue of instability in model performance by incorporating stability enhancements into the objective function.

Abstract

As mobile devices increasingly become focal points for advanced applications, edge computing presents a viable solution to their inherent computational limitations, particularly in deploying large language models (LLMs). However, despite the advancements in edge computing, significant challenges remain in efficient training and deploying LLMs due to the computational demands and data privacy concerns associated with these models. This paper explores a collaborative training framework that integrates mobile users with edge servers to optimize resource allocation, thereby enhancing both performance and efficiency. Our approach leverages parameter-efficient fine-tuning (PEFT) methods, allowing mobile users to adjust the initial layers of the LLM while edge servers handle the more demanding latter layers. Specifically, we formulate a multi-objective optimization problem to minimize the total energy consumption and delay during training. We also address the common issue of instability in model performance by incorporating stability enhancements into our objective function. Through novel fractional programming technique, we achieve a stationary point for the formulated problem. Simulations demonstrate that our method reduces the energy consumption as well as the latency, and increases the reliability of LLMs across various mobile settings.

Resource Allocation for Stable LLM Training in Mobile Edge Computing

TL;DR

A collaborative training framework that integrates mobile users with edge servers to optimize resource allocation, thereby enhancing both performance and efficiency and addressing the common issue of instability in model performance by incorporating stability enhancements into the objective function.

Abstract

As mobile devices increasingly become focal points for advanced applications, edge computing presents a viable solution to their inherent computational limitations, particularly in deploying large language models (LLMs). However, despite the advancements in edge computing, significant challenges remain in efficient training and deploying LLMs due to the computational demands and data privacy concerns associated with these models. This paper explores a collaborative training framework that integrates mobile users with edge servers to optimize resource allocation, thereby enhancing both performance and efficiency. Our approach leverages parameter-efficient fine-tuning (PEFT) methods, allowing mobile users to adjust the initial layers of the LLM while edge servers handle the more demanding latter layers. Specifically, we formulate a multi-objective optimization problem to minimize the total energy consumption and delay during training. We also address the common issue of instability in model performance by incorporating stability enhancements into our objective function. Through novel fractional programming technique, we achieve a stationary point for the formulated problem. Simulations demonstrate that our method reduces the energy consumption as well as the latency, and increases the reliability of LLMs across various mobile settings.
Paper Structure (16 sections, 3 theorems, 58 equations, 5 figures, 1 table)

This paper contains 16 sections, 3 theorems, 58 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Model
  4. Local Computation Model
  5. Edge Computation Model
  6. LLM Stability
  7. Problem Formulation
  8. Proposed Algorithm
  9. Optimizing $\boldsymbol{\alpha,p,b,f^U,f^E}$ given $\boldsymbol{\chi}$
  10. Optimizing $\boldsymbol{\chi}$ given $\boldsymbol{\alpha,p,b,f^U,f^E}$
  11. Simulations
  12. The performance of the proposed collaborative training method
  13. The performance of proposed algorithms under weighting factors
  14. The impact of the number of users and the number of edge servers
  15. Conclusion
  16. ...and 1 more sections

Key Result

Theorem 1

If a user fine-tunes a proportion $\alpha$ of the parameters, the expectation of the loss has an AS bounded by $\frac{2L^2}{k(1-\alpha)}$. I.e., $\forall i \in \{1, \ldots, k\},$

Figures (5)

  • Figure 1: The proposed system model consists of $N$ mobile users and $M$ edge servers. Our optimization problem aims to minimize energy consumption and delay while improving the LLM stability.
  • Figure 2: Comparison of system performance with and without the proposed collaborative training approach.
  • Figure 3: The performance of the proposed method under different weighting factors.
  • Figure 4: The convergence performance with different numbers of edge servers.
  • Figure 5: The performance of the algorithm under different numbers of mobile users.

Theorems & Definitions (7)

  • Definition 1: Average-replace-one stability
  • Theorem 1
  • proof
  • Proposition 1
  • proof
  • Lemma 1
  • proof