Transducing Language Models

TL;DR

Algorithms that compose a language model with an FST to *marginalize* over source strings mapping to a given target, propagating probabilities through the transducer without altering model parameters and enabling *conditioning* on transformed outputs are developed.

Abstract

Modern language models define distributions over strings, but downstream tasks often require different output formats. For instance, a model that generates byte-pair strings does not directly produce word-level predictions, and a DNA model does not directly produce amino-acid sequences. In such cases, a deterministic string-to-string transformation can convert the model's output to the desired form. This is a familiar pattern in probability theory: applying a function $f$ to a random variable $X\sim p$ yields a transformed random variable $f(X)$ with an induced distribution. While such transformations are occasionally used in language modeling, prior work does not treat them as yielding new, fully functional language models. We formalize this perspective and introduce a general framework for language models derived from deterministic string-to-string transformations. We focus on transformations representable as finite-state transducers -- a commonly used state-machine abstraction for efficient string-to-string mappings. We develop algorithms that compose a language model with an FST to *marginalize* over source strings mapping to a given target, propagating probabilities through the transducer without altering model parameters and enabling *conditioning* on transformed outputs. We present an exact algorithm, an efficient approximation, and a theoretical analysis. We conduct experiments in three domains: converting language models from tokens to bytes, from tokens to words, and from DNA to amino acids. These experiments demonstrate inference-time adaptation of pretrained language models to match application-specific output requirements.

Figures (21)

• Figure 1: GPT-2 probabilities for the BPE token strings that, when lowercased, match hello. The total probability of all such sequences is at the bottom.
• Figure 2: Decomposition algorithm.
• Figure 3: State-based instantiation of the checks from \ref{['alg:decompose']} on the precover DFA ${\color{TokenColor}\texttt{P}}_{\color{CharacterColor}\boldsymbol{y}}$. Helper functions $\hbox{\color{blue}step}$ and $\hbox{\color{blue}run}$ advance through the DFA; $\hbox{\color{blue}is_member}$ and $\hbox{\color{blue}is_live}$ are single-state lookups; $\hbox{\color{blue}is_cylinder}$ performs a BFS to verify universality.
• Figure 4: An FST for converting a token model into a character model. Paths for ␣cat and Dog.
• Figure 5: An FST that inserts a separator (sep) before commas followed by non-digit characters.

Theorems & Definitions (18)

• Example 1
• Example 2
• Theorem 5.1: Correctness of $\hbox{\color{blue}decompose}$
• Proposition 6.1
• Lemma 6.1
• Proposition C.1: Frontier containment
• Proposition C.2
• Proposition C.3: Non-cylinder monotonicity
• Proposition C.4: Cylinder uniqueness
• Proposition C.5: Combined universality