Token Sugar: Making Source Code Sweeter for LLMs through Token-Efficient Shorthand

TL;DR

Token Sugar introduces a reversible, token-efficient shorthand system to replace frequent, verbose semantic code patterns, addressing token costs beyond syntactic simplifications. It mines 799 sugarizable patterns from real-world Python code using generalized ASTs, encodes them with unique tokens, and trains LLMs on sugarized data to achieve substantial token reductions (up to 15.1%) while preserving generation quality (Pass@1 near baseline). When combined with SimPy, token savings exceed 20% in both LeetCode and HumanEval tasks, demonstrating complementary benefits. The approach demonstrates practical feasibility with three mainstream LLMs and outlines pathways for broader language support and more efficient training, highlighting its potential impact on production code-generation pipelines.

Abstract

Large language models (LLMs) have shown exceptional performance in code generation and understanding tasks, yet their high computational costs hinder broader adoption. One important factor is the inherent verbosity of programming languages, such as unnecessary formatting elements and lengthy boilerplate code. This leads to inflated token counts in both input and generated outputs, which increases inference costs and slows down the generation process. Prior work improves this through simplifying programming language grammar, reducing token usage across both code understanding and generation tasks. However, it is confined to syntactic transformations, leaving significant opportunities for token reduction unrealized at the semantic level. In this work, we propose Token Sugar, a concept that replaces frequent and verbose code patterns with reversible, token-efficient shorthand in the source code. To realize this concept in practice, we designed a systematic solution that mines high-frequency, token-heavy patterns from a code corpus, maps each to a unique shorthand, and integrates them into LLM pretraining via code transformation. With this solution, we obtain 799 (code pattern, shorthand) pairs, which can reduce up to 15.1% token count in the source code and is complementary to existing syntax-focused methods. We further trained three widely used LLMs on Token Sugar-augmented data. Experimental results show that these models not only achieve significant token savings (up to 11.2% reduction) during generation but also maintain near-identical Pass@1 scores compared to baselines trained on unprocessed code.

Figures (4)

• Figure 1: A demonstration of how Token Sugar facilitates LLMs' code generation process. The LLM equipped with Token Sugar can generate a compact, sugarized representation of the code using only 23 tokens, compared to the 40 tokens required by a vanilla LLM for the same logic.
• Figure 2: The conceptual relationship between the code pattern and its shorthand. The code snippet on the left costs 19 tokens measured by the tokenizer of GPT-4, but only 5 tokens when sugarized into our shorthand on the right.
• Figure 3: An example of how a standard AST is generalized. Shadowed nodes are replaced during generalization.
• Figure 4: An overview of the mining pipeline.