N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition

TL;DR

An early injection of the n-gram into the transformer decoder so that the network learns to fully leverage text-only data at the low additional cost of n-gram inference, which significantly reduces the performance gap between source and target corpora.

Abstract

Transformer-based encoder-decoder networks have recently achieved impressive results in handwritten text recognition, partly thanks to their auto-regressive decoder which implicitly learns a language model. However, such networks suffer from a large performance drop when evaluated on a target corpus whose language distribution is shifted from the source text seen during training. To retain recognition accuracy despite this language shift, we propose an external n-gram injection (NGI) for dynamic adaptation of the network's language modeling at inference time. Our method allows switching to an n-gram language model estimated on a corpus close to the target distribution, therefore mitigating bias without any extra training on target image-text pairs. We opt for an early injection of the n-gram into the transformer decoder so that the network learns to fully leverage text-only data at the low additional cost of n-gram inference. Experiments on three handwritten datasets demonstrate that the proposed NGI significantly reduces the performance gap between source and target corpora.

Figures (2)

• Figure 1: $n$-gram injection (NGI) into the auto-regressive decoder of the word attention network (WAN). An $n$-gram distribution vector $\bm{S}^\text{NGI}$ is made noisy by function $\phi$, projected by $f$ then summed with standard embeddings of previously output characters $\xi(\bm{c})$ and positional encoding $\bm{P}$ to form the new decoder input $\bm{X}$. Switching to an appropriate $n$-gram at test time reduces the error gap on target corpora.
• Figure 2: Scans of names (left) and surnames (right) from the N2S dataset.