Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revolutionizing Mixed Precision Quantization: Towards Training-free Automatic Proxy Discovery via Large Language Models

Haidong Kang, Jun Du, Lihong Lin

TL;DR

This paper tackles the bottleneck of designing effective proxies for mixed-precision quantization (MPQ) by introducing TAP, an LLM-driven, training-free proxy discovery framework. TAP uses a three-component system (proxy candidate generator, fitness evaluator, and a Direct Policy Optimization loop) to automatically evolve high-quality MPQ proxies without human expertise or training. Across CNNs and Transformers, TAP achieves state-of-the-art accuracy with microsecond-scale search times and strong generalization to PTQ and larger architectures. The work demonstrates the viability of leveraging large language models for hardware-aware design tasks, offering a scalable path for MPQ across diverse models and datasets.

Abstract

Mixed-Precision Quantization (MPQ) liberates the Deep Neural Networks (DNNs) from the Out-Of-Memory (OOM) bottleneck, which garnered increasing research attention. However, conventional methods either searched from costly differentiable optimization, which is neither efficient nor flexible, or learned a quantized DNN from the proxy (i.e., HAWQ) manually designed by human experts, which is labor-intensive and requires huge expert knowledge. Can we design a proxy without involving any human experts and training? In this paper, we provide an affirmative answer by proposing a novel Large Language Models (LLMs)-driven Training-free Automatic Proxy (dubbed TAP) discovery framework, which reforms the design paradigm of MPQ by utilizing LLMs to find superior TAP tailored for MPQ, automatically. In addition, to bridge the gap between black-box LLMs and the tough MPQ task, we ingeniously propose simple Direct Policy Optimization (DPO) based reinforcement learning to enhance LLMs' reasoning by optimizing prompts, which can construct a positive feedback loop between the LLM and the MPQ task, enabling LLMs to generate better TAP in the next evolution. Extensive experiments on mainstream benchmarks demonstrate that TAP achieves state-of-the-art performance. Finally, we truly believe that our TAP will significantly contribute to the MPQ community by providing a new perspective on LLM-driven design algorithms.

Revolutionizing Mixed Precision Quantization: Towards Training-free Automatic Proxy Discovery via Large Language Models

TL;DR

This paper tackles the bottleneck of designing effective proxies for mixed-precision quantization (MPQ) by introducing TAP, an LLM-driven, training-free proxy discovery framework. TAP uses a three-component system (proxy candidate generator, fitness evaluator, and a Direct Policy Optimization loop) to automatically evolve high-quality MPQ proxies without human expertise or training. Across CNNs and Transformers, TAP achieves state-of-the-art accuracy with microsecond-scale search times and strong generalization to PTQ and larger architectures. The work demonstrates the viability of leveraging large language models for hardware-aware design tasks, offering a scalable path for MPQ across diverse models and datasets.

Abstract

Mixed-Precision Quantization (MPQ) liberates the Deep Neural Networks (DNNs) from the Out-Of-Memory (OOM) bottleneck, which garnered increasing research attention. However, conventional methods either searched from costly differentiable optimization, which is neither efficient nor flexible, or learned a quantized DNN from the proxy (i.e., HAWQ) manually designed by human experts, which is labor-intensive and requires huge expert knowledge. Can we design a proxy without involving any human experts and training? In this paper, we provide an affirmative answer by proposing a novel Large Language Models (LLMs)-driven Training-free Automatic Proxy (dubbed TAP) discovery framework, which reforms the design paradigm of MPQ by utilizing LLMs to find superior TAP tailored for MPQ, automatically. In addition, to bridge the gap between black-box LLMs and the tough MPQ task, we ingeniously propose simple Direct Policy Optimization (DPO) based reinforcement learning to enhance LLMs' reasoning by optimizing prompts, which can construct a positive feedback loop between the LLM and the MPQ task, enabling LLMs to generate better TAP in the next evolution. Extensive experiments on mainstream benchmarks demonstrate that TAP achieves state-of-the-art performance. Finally, we truly believe that our TAP will significantly contribute to the MPQ community by providing a new perspective on LLM-driven design algorithms.

Paper Structure

This paper contains 17 sections, 7 equations, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. PRELIMINARIES and Motivations
  3. Rethinking the Training-free MPQ
  4. Observations and Motivations
  5. Approach
  6. Overview of TAP
  7. Automatic Proxy Discovery for Mixed-Precision Quantization
  8. Evolutionary Framework of the TAP
  9. Experiments
  10. Quantization-Aware Training
  11. Post-Training Quantization
  12. Generalization of TAP to Larger Transformer
  13. Efficiency of TAP
  14. Ablation studies
  15. The impact of various LLMs:
  16. ...and 2 more sections

Figures (2)

  • Figure 1: TAPvs.Peer competitors in terms of accuracy, generalization, and costs.
  • Figure 2: Overview of TAP. Input the initial prompt first. The LLM generates an initial proxy population, where each proxy must include inference logic, executable code, and bit-width allocation. Candidate proxies are validated on benchmarks like ImageNet-1k, with quantization accuracy and computational cost calculated to produce an adaptability score. Using this score, construct proxy preference data pairs, optimize LLM parameters via the DPO module to guide subsequent proxy generation, merge the current and candidate populations, and filter by adaptability score to maintain a fixed size.