ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model

TL;DR

This work proposes replacing and reinitializing the parameters of the model's input and output layers with the parameters of the original model, and training these parameters while keeping other parameters fixed.

Abstract

Tokenizer is an essential component for large language models (LLMs), and a tokenizer with a high compression rate can improve the model's representation and processing efficiency. However, the tokenizer cannot ensure high compression rate in all scenarios, and an increase in the average input and output lengths will increases the training and inference costs of the model. Therefore, it is crucial to find ways to improve the model's efficiency with minimal cost while maintaining the model's performance. In this work, we propose a method to improve model representation and processing efficiency by replacing the tokenizers of LLMs. We propose replacing and reinitializing the parameters of the model's input and output layers with the parameters of the original model, and training these parameters while keeping other parameters fixed. We conducted experiments on different LLMs, and the results show that our method can maintain the performance of the model after replacing the tokenizer, while significantly improving the decoding speed for long texts.

Figures (3)

• Figure 1: Illustration of the proposed method.
• Figure 2: Comparison of validation loss changes between ReTok and random initialization based on Llama3-8B.
• Figure 3: Comparison of decoding speeds based on the Llama3-8B model on the Chinese validation set for texts of different lengths.