From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs

TL;DR

HAQA targets the challenge of deploying large language models on resource-constrained hardware without sacrificing accuracy. It uses an LLM-based agent with a dual-prompt design to jointly optimize quantization hyperparameters and hardware deployment settings, including bit-widths and kernel configurations. The approach demonstrates up to 2.3x inference speedups on LLaMA2-7B and robust accuracy gains across multiple models and devices, including mobile hardware, while adapting to diverse architectures. This framework lowers manual effort, broadens accessibility for non-experts, and provides a scalable path toward efficient edge deployment of LLMs.

Abstract

Deploying models, especially large language models (LLMs), is becoming increasingly attractive to a broader user base, including those without specialized expertise. However, due to the resource constraints of certain hardware, maintaining high accuracy with larger model while meeting the hardware requirements remains a significant challenge. Model quantization technique helps mitigate memory and compute bottlenecks, yet the added complexities of tuning and deploying quantized models further exacerbates these challenges, making the process unfriendly to most of the users. We introduce the Hardware-Aware Quantization Agent (HAQA), an automated framework that leverages LLMs to streamline the entire quantization and deployment process by enabling efficient hyperparameter tuning and hardware configuration, thereby simultaneously improving deployment quality and ease of use for a broad range of users. Our results demonstrate up to a 2.3x speedup in inference, along with increased throughput and improved accuracy compared to unoptimized models on Llama. Additionally, HAQA is designed to implement adaptive quantization strategies across diverse hardware platforms, as it automatically finds optimal settings even when they appear counterintuitive, thereby reducing extensive manual effort and demonstrating superior adaptability. Code will be released.

Figures (5)

• Figure 1: Comparison of human-driven (a) and agent-based (b) optimization for AI model deployment. Deploying an LLM on-device usually involves quantizing the model to lower bit precision to reduce memory consumption and tailoring deployment strategies for each hardware platform—steps that require repeated expert tuning and still may fail to deliver optimal performance. With more non-expert users wanting to run LLMs locally, manual adjustment of quantization parameters and deployment configurations becomes impractical. We propose an LLM-based quantization agent to automate and streamline the entire workflow, enabling efficient hyperparameter search and hardware-specific configuration. Unlike experts who tweak one parameter at a time, our agent jointly tunes all settings for synergistic optimization and consistent performance.
• Figure 2: This Figure illustrates a sample prompt, demonstrating their practical structure. Blue text highlights key information conveyed to the agent. Purple text is used to enhance agent's decision-making ability, which we'll discuss in Section \ref{['sec:friendly']}. The full prompt and detailed descriptions of each component is in Appendix \ref{['Appendix_1']}. Panels (a)–(c) denote the static prompt’s components: (a) hardware platform information, (b) deployment objective details , and (c) fine-tuning objective details, while panel (d) represents the dynamic prompt, which contains logs and conversation history.
• Figure 3: To enable the LLM-based agent, we first design a prompt with two parts: the static and dynamic prompts. The static prompt includes configurations that remain unchanged, such as memory limits, platform specifications, hardware details, and quantization algorithm hyperparameters. The dynamic prompt contains information that changes dynamically, such as hardware speed testing logs and quantization result logs. During the quantization and deployment stage, the agent generates optimization guidelines based on the current prompt. The quantization algorithm and hardware deployment then execute based on the guidelines, providing accuracy and speed feedback. Using this feedback and the prompt history, the agent updates the dynamic part of the prompt and generates a new guideline.
• Figure 4: Convergence curve of HAQA and existing tuning approaches on multiple tasks.
• Figure 5: Token Generation Speed of Different Models Across Various Quantization Configurations (FP16, INT8, INT4).