Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning Transformers: Vocabulary Transfer

Vladislav Mosin, Igor Samenko, Alexey Tikhonov, Borislav Kozlovskii, Ivan P. Yamshchikov

TL;DR

This work introduces vocabulary transfer as a transfer-learning paradigm for transformers, formalizing corpus-specific tokenization and embedding initialization as a means to improve fine-tuning. The proposed VIPI method initializes new vocabulary embeddings by partial inheritance from the pretrained space, preserving information via partition-based averaging. Across multiple datasets (Quora, Sentiment140, Hyperpartisan News) and vocabulary sizes, corpus-specific tokenization plus VIPI, especially with an intermediary MLM fine-tuning step, yields consistent accuracy gains and faster transfer, with notable benefits in scientific text processing (MeDAL). The findings suggest expanding transfer learning to include tokenization adaptation, with practical impact for domain-specific and specialized-language NLP tasks.

Abstract

Transformers are responsible for the vast majority of recent advances in natural language processing. The majority of practical natural language processing applications of these models are typically enabled through transfer learning. This paper studies if corpus-specific tokenization used for fine-tuning improves the resulting performance of the model. Through a series of experiments, we demonstrate that such tokenization combined with the initialization and fine-tuning strategy for the vocabulary tokens speeds up the transfer and boosts the performance of the fine-tuned model. We call this aspect of transfer facilitation vocabulary transfer.

Fine-Tuning Transformers: Vocabulary Transfer

TL;DR

This work introduces vocabulary transfer as a transfer-learning paradigm for transformers, formalizing corpus-specific tokenization and embedding initialization as a means to improve fine-tuning. The proposed VIPI method initializes new vocabulary embeddings by partial inheritance from the pretrained space, preserving information via partition-based averaging. Across multiple datasets (Quora, Sentiment140, Hyperpartisan News) and vocabulary sizes, corpus-specific tokenization plus VIPI, especially with an intermediary MLM fine-tuning step, yields consistent accuracy gains and faster transfer, with notable benefits in scientific text processing (MeDAL). The findings suggest expanding transfer learning to include tokenization adaptation, with practical impact for domain-specific and specialized-language NLP tasks.

Abstract

Transformers are responsible for the vast majority of recent advances in natural language processing. The majority of practical natural language processing applications of these models are typically enabled through transfer learning. This paper studies if corpus-specific tokenization used for fine-tuning improves the resulting performance of the model. Through a series of experiments, we demonstrate that such tokenization combined with the initialization and fine-tuning strategy for the vocabulary tokens speeds up the transfer and boosts the performance of the fine-tuned model. We call this aspect of transfer facilitation vocabulary transfer.
Paper Structure (14 sections, 4 figures, 9 tables)

This paper contains 14 sections, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vocabulary Transfer
  4. Example of Vocabulary Transfer: VIPI
  5. Data
  6. Experiments
  7. Aspects of Vocabulary Transfer
  8. Embeddings Initialization
  9. Fine-tuning Embeddings
  10. Experimenting with the Size of Vocabulary
  11. Experiments with different tokenizations
  12. Vocabulary Transfer for Scientific Text Processing
  13. Discussion
  14. Conclusion

Figures (4)

  • Figure 1: Quora insincere questions classifier epoch train loss. Fine-tuning the pretrained BERT with tokenization of various sizes. Corpus-specific tokenization combined with VIPI and a pretrained body speeds up learning. Models are ranked on the figure according to the resulting loss.
  • Figure 2: Hyperpartisan news classifier epoch train loss. Fine-tuning the pretrained BERT with tokenization of various sizes. The model with corpus-specific tokenizations and pretrained body learns faster than the model with original tokenization. Models are ranked on the figure according to the resulting loss.
  • Figure 3: Fine-tuning a BERT classifier over hyperpartisan news detection data for 32 000 tokens. The first plot corresponds to the original tokenization. The second plot shows the relative performance of a classifier trained from scratch with new tokenization. The intermediary MLM step hardly plays any role when one trains a model with new tokenization from scratch. The two last plots demonstrate that pre-training an MLM model with dataset-specific tokenization boosts the performance of the resulting classifier. The classifier without MLM is represented on every plot with a dashed line.
  • Figure 4: Relative information compression with data-specific tokenization for Quora insincere questions and hyperpartisan news datasets. The ratio between information estimators for dataset-specific tokenization and the same information estimator for original tokenization. Dataset-specific tokenization compresses hyperpartisan news datasets across various vocabulary sizes. Dataset-specific tokenization for Quora does not compress data for larger vocabularies and generally is weaker than for other datasets.