DILEMMA: Joint LLM Quantization and Distributed LLM Inference Over Edge Computing Systems

TL;DR

Deploying LLMs at the network edge is challenged by limited communication, computation, and storage resources. DILEMMA introduces a joint optimization framework that combines layer placement and layer-wise quantization, solved via an Integer Linear Programming formulation and enhanced by knowledge distillation to preserve accuracy. The approach is evaluated on the OPT-350 model with the SQuAD dataset, achieving up to a 12.75% quantization ratio while maintaining performance and reducing latency. This work enables practical, low-latency LLM inference in edge environments for smart-city applications, with potential for broad applicability in resource-constrained distributed inference.

Abstract

With a recent trend of using Large Language Models (LLMs) for different applications within smart cities, there is a need for pushing these models toward the edge of network while still preserving their performance. Edge Computing (EC) as a physically closer computing resource to the end users can help to reduce the communication delay for serving end users' tasks for LLM-dependent services. However, EC servers have limited capacity in terms of communication, computation, and storage capacity. This paper introduces DILEMMA, a novel framework addressing the challenges of deploying LLMs in EC systems by jointly optimizing layer placement and layer quantization in EC systems. DILEMMA formulates an Integer Linear Programming problem to minimize total inference delay while ensuring acceptable LLM performance levels, leveraging layer-wise quantization and knowledge distillation for LLM performance control. Experimental evaluations on OPT-350 model using the SQuAD dataset demonstrate that DILEMMA achieves a quantization ratio of up to 12.75% while preserving model loss, highlighting its effectiveness in resource-constrained environments.