Performance Trade-offs of Optimizing Small Language Models for E-Commerce

TL;DR

This work demonstrates that a 1B open-weight Llama 3.2 model can achieve $0.99$ exact-match accuracy on multilingual e-commerce intent extraction when fine-tuned with QLoRA on a high-quality synthetic dataset. Post-training quantization reveals hardware-dependent trade-offs: GPTQ dramatically reduces VRAM but can slow GPU inference on older architectures, whereas CPU-friendly GGUF enables up to $18\times$ speedups and >$90\%$ RAM reduction. The study provides a nuanced Pareto analysis showing that 5-bit GGUF preserves accuracy while 4-bit GGUF maximizes CPU throughput, and that aggressive 3-bit quantization can be catastrophic. Overall, the results advocate a hardware-aware, many-small-model approach for domain-specific AI deployment, complemented by released synthetic data to support reproducibility.

Abstract

Large Language Models (LLMs) offer state-of-the-art performance in natural language understanding and generation tasks. However, the deployment of leading commercial models for specialized tasks, such as e-commerce, is often hindered by high computational costs, latency, and operational expenses. This paper investigates the viability of smaller, open-weight models as a resource-efficient alternative. We present a methodology for optimizing a one-billion-parameter Llama 3.2 model for multilingual e-commerce intent recognition. The model was fine-tuned using Quantized Low-Rank Adaptation (QLoRA) on a synthetically generated dataset designed to mimic real-world user queries. Subsequently, we applied post-training quantization techniques, creating GPU-optimized (GPTQ) and CPU-optimized (GGUF) versions. Our results demonstrate that the specialized 1B model achieves 99% accuracy, matching the performance of the significantly larger GPT-4.1 model. A detailed performance analysis revealed critical, hardware-dependent trade-offs: while 4-bit GPTQ reduced VRAM usage by 41%, it paradoxically slowed inference by 82% on an older GPU architecture (NVIDIA T4) due to dequantization overhead. Conversely, GGUF formats on a CPU achieved a speedup of up to 18x in inference throughput and a reduction of over 90% in RAM consumption compared to the FP16 baseline. We conclude that small, properly optimized open-weight models are not just a viable but a more suitable alternative for domain-specific applications, offering state-of-the-art accuracy at a fraction of the computational cost.

Figures (9)

• Figure 1: Overall Model Performance Comparison. The fine-tuned Llama 3.2 1B model and its high-quality quantized variants achieve accuracy scores comparable to the top commercial baselines.
• Figure 2: GPU Memory Comparison. GPTQ 4-bit quantization significantly reduces total VRAM, peak VRAM, and the memory required for model parameters alone.
• Figure 3: GPU speed profile on NVIDIA T4. GPTQ loads much faster but runs with lower throughput.
• Figure 4: Energy per token on NVIDIA T4. GPTQ consumes substantially more energy per generated token.
• Figure 5: CPU Inference Performance. All GGUF-quantized models offer a dramatic speedup over the full-precision baseline.