New Solutions on LLM Acceleration, Optimization, and Application

TL;DR

The paper addresses the efficiency gap of large language models (LLMs) by surveying and proposing solutions across algorithmic, hardware, compiler, and design-automation dimensions. It highlights concrete techniques such as Medusa for parallel decoding, SnapKV for KV-cache compression, AutoDistill for hardware-aware model compression, and HLS-based compilers ScaleHLS and HIDA that map models to accelerators. A hardware-aware and edge-conscious perspective is integrated, alongside a case study in EDA leveraging the Chrysalis dataset and an HLS debugging assistant to accelerate hardware design verification. Collectively, these contributions enable faster, more memory-efficient LLM deployment and open new avenues for LLM applications in hardware design, verification, and edge computing, with future directions spanning system-aware optimization, reconfigurable hardware, and automated formal verification and design automation.

Abstract

Large Language Models (LLMs) have become extremely potent instruments with exceptional capacities for comprehending and producing human-like text in a wide range of applications. However, the increasing size and complexity of LLMs present significant challenges in both training and deployment, leading to substantial computational and storage costs as well as heightened energy consumption. In this paper, we provide a review of recent advancements and research directions aimed at addressing these challenges and enhancing the efficiency of LLM-based systems. We begin by discussing algorithm-level acceleration techniques focused on optimizing LLM inference speed and resource utilization. We also explore LLM-hardware co-design strategies with a vision to improve system efficiency by tailoring hardware architectures to LLM requirements. Further, we delve into LLM-to-accelerator compilation approaches, which involve customizing hardware accelerators for efficient LLM deployment. Finally, as a case study to leverage LLMs for assisting circuit design, we examine LLM-aided design methodologies for an important task: High-Level Synthesis (HLS) functional verification, by creating a new dataset that contains a large number of buggy and bug-free codes, which can be essential for training LLMs to specialize on HLS verification and debugging. For each aspect mentioned above, we begin with a detailed background study, followed by the presentation of several novel solutions proposed to overcome specific challenges. We then outline future research directions to drive further advancements. Through these efforts, we aim to pave the way for more efficient and scalable deployment of LLMs across a diverse range of applications.

Figures (9)

• Figure 1: The proposed parallel decoding framework Medusa. During inference, each head generates multiple top predictions for its designated position. These predictions are assembled into candidates processed in parallel using a tree-based attention mechanism. Then the framework verifies the candidates and accepts a continuation cai2024medusa.
• Figure 2: The graph shows the simplified workflow of SnapKV, where the orange area represents the group of positions per head clustered and selected by SnapKV.
• Figure 3: The proposed AutoDistill framework zhang2022autodistill.
• Figure 4: The profiling results on the activity of heads across different datasets by measuring each head's contribution based on its variance over the input sequence. Heads that show low variance are considered inactive, leading to contextual sparsity.
• Figure 5: The preliminary result from forward throughput improvement. Flash2_hmask is the result from the combination of FlashAttention2 dao2022flashattention and our pruning-aware quantization approach wan2024software.