LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model

TL;DR

LoPT targets the tokenization bottleneck in long-context LLM inference by delivering lossless parallel tokenization. It introduces a three-module pipeline—text splitting, parallel tokenization, and position-aware merge—with a dynamic chunk-length mechanism to guarantee outputs identical to sequential tokenization. The authors provide a theoretical consistency proof for WordPiece and BPE tokenization and validate the approach across English and Chinese long-text datasets, achieving lossless accuracy with substantial speedups over existing delimiter- and overlap-based methods. This work enables efficient, scalable long-context inference by removing tokenization-induced latency without compromising output fidelity.

Abstract

Long context inference scenarios have become increasingly important for large language models, yet they introduce significant computational latency. While prior research has optimized long-sequence inference through operators, model architectures, and system frameworks, tokenization remains an overlooked bottleneck. Existing parallel tokenization methods accelerate processing through text segmentation and multi-process tokenization, but they suffer from inconsistent results due to boundary artifacts that occur after merging. To address this, we propose LoPT, a novel Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. Our approach employs character-position-based matching and dynamic chunk length adjustment to align and merge tokenized segments accurately. Extensive experiments across diverse long-text datasets demonstrate that LoPT achieves significant speedup while guaranteeing lossless tokenization. We also provide theoretical proof of consistency and comprehensive analytical studies to validate the robustness of our method.

Figures (8)

• Figure 1: This figure shows the error cases of the previous chunk-based parallel tokenization methods (delimiter-based and overlap-based) and how our method LoPT obtains the correct result. Their outputs are in blue, and the standard tokenization results without splitting are in green. For delimiter-based parallel tokenization, incorrect results occur due to the neglect of token changes caused by variations in segment boundaries during the merge phase. For the overlap-based methods, although they attempted to address this issue by identifying overlap token sequences in the tokenization results of two adjacent text chunks, incorrect results still appear due to the mismatch of the overlap token sequence, which often occurs when processing certain texts, such as those containing consecutive repetitions. The LoPT identifies overlap tokens based on the token character position, which helps avoid such a mismatch.
• Figure 2: The ParallelTokenizer's process to parallelly tokenize a long text. Here, we use two text segments as an example.
• Figure 3: This figure illustrates the workflow of our method, with the green sections representing our method modules. The input long text is first split by the text split module into text chunks of chunk length, with adjacent text chunks having an overlap of overlap length. Then, each text chunk is processed by the parallel tokenization module to generate the corresponding token sequence. Unlike previous methods, at this step, we also output the character-level position of each token within the corresponding text chunk, providing the necessary positional information for merging the tokenization results. Finally, the token merge module receives the tokens and token position information of each text chunk and performs the merging operation.
• Figure 4: A performance comparison between the HuggingFace TokenizerFast ("·" points) and our framework LoPT ("*" points) on the LongBenchV2 dataset, with different sequence lengths. The horizontal axis represents the sequence length (number of tokens), and the vertical axis represents the tokenization time consumption (ms).
• Figure 5: Performance of tokenization using our framework on LongBenchV2 and ClongEval datasets with different chunk lengths. The process pool size is 32. The horizontal axis represents the chunk length (character-level), and the vertical axis represents the tokenization time consumption. The dashed line represents the average number of chunks corresponding to different chunk lengths.

Theorems & Definitions (6)

• Theorem 4.1
• Lemma 7.1
• Lemma 7.2
• Lemma 7.3
• proof
• proof