LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi

TL;DR

This study demonstrates that aggressive quantization strategies can significantly reduce energy consumption while maintaining inference quality, making LLMs practical for resource-limited environments.

Abstract

Deploying Large Language Models (LLMs) on resource-constrained edge devices like the Raspberry Pi presents challenges in computational efficiency, power consumption, and response latency. This paper explores quantization-based optimization techniques to enable high-throughput, energy-efficient execution of LLMs on low-power embedded systems. Our approach leverages k-quantization, a Post-Training Quantization (PTQ) method designed for different bit-widths, enabling efficient 2-bit, 4-bit, 6-bit, and 8-bit weight quantization. Additionally, we employ ternary quantization using Quantization-Aware Training (QAT) for BitNet models, allowing for more effective adaptation to lower-bit representations while preserving accuracy. Our findings highlight the potential of quantized LLMs for real-time conversational AI on edge devices, paving the way for low-power, high-efficiency AI deployment in mobile and embedded applications. This study demonstrates that aggressive quantization strategies can significantly reduce energy consumption while maintaining inference quality, making LLMs practical for resource-limited environments.

Figures (7)

• Figure 1: The typical architecture of decoder-only LLM models and its underlying operations.
• Figure 2: TPS across different LLMs and precision levels.
• Figure 3: Energy efficiency of LLMs measured in TPJ.
• Figure 4: Words per Battery Life (W/BL) for different LLMs and precision levels.
• Figure 5: NUBIA score for different LLMs and precision levels.