DocTTT: Test-Time Training for Handwritten Document Recognition Using Meta-Auxiliary Learning

TL;DR

DocTTT tackles handwritten document recognition by enabling test-time adaptation of the visual backbone through a self-supervised Masked Autoencoder loss, tailored per input to handle diverse handwriting and layouts. It combines model-agnostic meta-learning with a MAE auxiliary task, forming a bi-level optimization where inner updates adapt to each test instance and outer updates optimize overall HDR performance. The approach is built on a two-branch architecture sharing a backbone, with Phase 1 pre-training on synthetic data and Phase 2 meta-training with curriculum from synthetic to real data. Empirical results on READ2016, IAM, and RIMES demonstrate consistent gains over prior methods in both text and layout metrics, with ablations confirming the necessity of each component. Overall, DocTTT provides a practical, adaptable HDR solution that can quickly specialize to new documents without requiring extra annotations at test time.

Abstract

Despite recent significant advancements in Handwritten Document Recognition (HDR), the efficient and accurate recognition of text against complex backgrounds, diverse handwriting styles, and varying document layouts remains a practical challenge. Moreover, this issue is seldom addressed in academic research, particularly in scenarios with minimal annotated data available. In this paper, we introduce the DocTTT framework to address these challenges. The key innovation of our approach is that it uses test-time training to adapt the model to each specific input during testing. We propose a novel Meta-Auxiliary learning approach that combines Meta-learning and self-supervised Masked Autoencoder~(MAE). During testing, we adapt the visual representation parameters using a self-supervised MAE loss. During training, we learn the model parameters using a meta-learning framework, so that the model parameters are learned to adapt to a new input effectively. Experimental results show that our proposed method significantly outperforms existing state-of-the-art approaches on benchmark datasets.

Figures (7)

• Figure 1: Illustration of the handwritten document recognition (HDR) problem. Given an input image (left) of a handwritten document, the output of HDR is an XML file (right) describing the page layout. The XML-based representation of the page captures the layout and text tokens of the document image albahli2021improvednurseitov2021handwrittencoquenet2023dan. The XML representation can be flattened as a sequence of tokens. The goal of our work is to predict this sequence of tokens for a given document image.
• Figure 2: Illustration of the capability of Masked Autoencoders. MAE effectively tackles the HDR problem by reconstructing diverse handwriting styles from unseen test data. The image features four distinct writing styles (first column), a 75% masked image (second column), and reconstructions from both low-resolution (Reconstructed A) and high-resolution (Reconstructed B) inputs. Accurately predicting token sequences from these varied handwriting styles in document images presents substantial challenges. This illustrates the need for HDR solutions that can adapt to different writing styles.
• Figure 3: Illustration of our model architecture. On the left, the input document image is masked and then passes through shared weights to the auxiliary branch for the reconstruction task. We use the auxiliary branch to fine-tune the model for each instance using a self-supervised masked autoencoder loss. The adapted model is then used for the primary task of predicting the flattened XML representation (see Figure \ref{['inputoutput']}) as a sequence of tokens.
• Figure 4: Illustration of the Meta-Auxiliary training. For a training example $\mathbf{x}_i$ with the ground-truth label $\mathbf{y}_i$, we update the model parameters $\Theta$ using a self-supervised MAE loss (inner loop) to obtain an adapted model $\Theta'$ for this training example. We then use the adapted model $\Theta'$ for the primary task. We use the supervised loss for the primary task as the meta-objective in the outer loop to update the model parameter $\Theta$.
• Figure 5: Illustration of the auxiliary branch. Given an input image (top left), we first generate a masked version (top middle) of the input image. The output of the auxiliary branch outputs a reconstruction (top right) of the input image based on its masked version. The bottom two images show the zoomed-in regions of the input and the reconstruction images.