DVAGen: Dynamic Vocabulary Augmented Generation

TL;DR

DVAGen addresses the fixed-vocabulary bottleneck in language models by introducing a dynamic vocabulary augmentation framework that integrates a phrase encoder with a standard LM. It provides a modular, open-source pipeline (DVAModel, PhraseSampler, DVATokenizer) with CLI and WebUI for training, inference, and visualization, enabling per-sentence dynamic vocabularies and batch inference for throughput gains. The approach expands the vocabulary by attaching per-sentence phrase embeddings and computing logits over $|V|+|P|$, guided by retrieval-augmented phrase sampling via a FAISS-based retriever. Empirical results on modern LLMs show improved generation quality (MAUVE, Diversity, PPL) and higher inference throughput, validating the framework’s practical value and memory-efficient training with frozen backbones. Overall, DVAGen offers a scalable, plug-and-play solution for dynamic vocabulary-augmented language modeling that supports end-to-end training, inference, and visualization.

Abstract

Language models trained with a fixed vocabulary struggle to generalize to novel or out-of-vocabulary words, limiting their flexibility in handling diverse token combinations. Existing dynamic vocabulary approaches attempt to address this limitation but face challenges such as fragmented codebases, lack of support for modern LLMs, and limited inference scalability. To overcome these issues, we introduce DVAGen, a fully open-source, unified framework designed for training, evaluation, and visualization of dynamic vocabulary-augmented language models. Our framework modularizes the pipeline for ease of customization, integrates seamlessly with open-source LLMs, and is the first to provide both CLI and WebUI tools for real-time result inspection. We validate the effectiveness of dynamic vocabulary methods on modern LLMs and demonstrate support for batch inference, significantly improving inference throughput.

Figures (8)

• Figure 1: The overall architecture of DVAGen. The light color boxes represent modules, while the dashed boxes indicate collections of modules with relevant functions.
• Figure 2: The overall training pipline of DVAGen.
• Figure 3: The overall inference pipline of DVAGen.
• Figure 4: The screenshot of the DVAGen WebUI interface.
• Figure 5: The screenshot displays the generation outputs. Phrases are highlighted in red, and the logit probabilities for each token and phrase can be viewed by clicking on the respective token or phrase.