LLM-based Affective Text Generation Quality Based on Different Quantization Values
Yarik Menchaca Resendiz, Roman Klinger
TL;DR
This work quantifies the resource-efficiency trade-offs of quantizing large language models for affective text generation. By evaluating 8-, 16-, and 32-bit configurations across multiple open-weight models, it reports memory savings of approximately $76\%$ with up to a $\pm$ $F_1$-score change dependent on model size, and longer inference times at lower precision. Larger models tend to lose more $F_1$ performance when aggressively quantized, while smaller models can maintain or even improve relative quality, with text quality favoring larger models at lower quantization. The findings inform deployment decisions for resource-constrained environments, balancing memory, speed, and affective text generation quality, and point to future work in broader tasks, languages, and quantization-aware training.
Abstract
Large language models exhibit a remarkable capacity in language generation and comprehension. These advances enable AI systems to produce more human-like and emotionally engaging text. However, these models rely on a large number of parameters, requiring significant computational resources for training and inference. In some scenarios, accessing these resources can be challenging (e.g., budget or hardware limitations). Techniques like reducing precision bits can make models more memory-efficient, reducing the computational resources needed, at the cost of reduced accuracy. This paper addresses the trade-off between different quantization values, GPU RAM utilization, and text quality in affective text generation (e.g., "I really enjoy running in the snow-covered forest"). To evaluate, we use an emotion classifier and ten seed prompts to generate affective text. We test three setups of precision bits (8, 16, and 32) across five open-weight language models from two different families. Our findings demonstrate that bit reductions lead to memory savings, achieving a reduction of 76%. However, this optimization comes with a trade-off, leading to a decrease of up to 10 pp in F1 score for larger models and an increase of 10 pp for smaller models, along with roughly double the inference time. In terms of text quality, larger models at lower quantization levels generally outperform smaller, higher-precision models -- while requiring similar memory.