Optimizing LLMs Using Quantization for Mobile Execution

TL;DR

The paper investigates on-device deployment of large language models by applying 4-bit post-training quantization to Llama 3.2 3B and validating the workflow on Android. It uses BitsAndBytes nf4 quantization, GGUF conversion via llama.cpp, and Termux/Ollama to demonstrate mobile inference with a 68.66% reduction in model size. While qualitative testing shows coherent outputs, the study also compares MMLU performance with other edge models to assess trade-offs between size and accuracy. The work highlights PTQ with mobile-optimized formats as a practical path toward private, offline LLM use on commodity devices.

Abstract

Large Language Models (LLMs) offer powerful capabilities, but their significant size and computational requirements hinder deployment on resource-constrained mobile devices. This paper investigates Post-Training Quantization (PTQ) for compressing LLMs for mobile execution. We apply 4-bit PTQ using the BitsAndBytes library with the Hugging Face Transformers framework to Meta's Llama 3.2 3B model. The quantized model is converted to GGUF format using llama.cpp tools for optimized mobile inference. The PTQ workflow achieves a 68.66% reduction in model size through 4-bit quantization, enabling the Llama 3.2 3B model to run efficiently on an Android device. Qualitative validation shows that the 4-bit quantized model can perform inference tasks successfully. We demonstrate the feasibility of running the quantized GGUF model on an Android device using the Termux environment and the Ollama framework. PTQ, especially at 4-bit precision combined with mobile-optimized formats like GGUF, provides a practical pathway for deploying capable LLMs on mobile devices, balancing model size and performance.