Compressed code: the hidden effects of quantization and distillation on programming tokens
Viacheslav Siniaev, Iaroslav Chelombitko, Aleksey Komissarov
TL;DR
This study addresses how programming languages are represented in LLM tokenizers and how compression techniques—quantization, distillation, and model scaling—alter token-level coding behavior. It introduces a cold-start probability framework (PKP, STP, KAP, NLP) to evaluate coding abilities without prompts and analyzes keyword coverage, token ranking, and language-marker representation across multiple tokenizers. The findings reveal that code-specialized models do not exhibit fundamentally different token distributions from general models, yet compression can produce non-linear shifts in keyword and special-token probabilities, with distillation tending to reduce semantic tokens while increasing structural tokens. The work provides empirical guidelines for maintaining code-generation quality under resource constraints and offers a practical framework for assessing token-level impacts of model optimization in production environments.
Abstract
Large Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming language token representations, we characterize how programming languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization, distillation, model scaling, and task-specific fine-tuning - affect token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments.