Align, Minimize and Diversify: A Source-Free Unsupervised Domain Adaptation Method for Handwritten Text Recognition

TL;DR

This work tackles handwritten text recognition under domain shift without access to source data by introducing Align, Minimize and Diversify (AMD), a three-term loss that leverages BN statistics for distribution alignment, drives confident frame-wise predictions, and preserves diversity across target sequences. A two-stage process pre-trains on a source dataset and then adapts using only unlabeled target images, with final loss $\,\mathcal{L} = w_a L_a + w_m L_m + w_d L_d$ guiding the optimization. Evaluations on IAM, GW, ESPOSALLES and synthetic data show AMD is competitive with DA methods that rely on source data, particularly excelling in OOD conditions and synthetic setups; limitations include reliance on BN for alignment and absence of language-model adaptation. The results highlight AMD as a practical SFUDA approach for HTR, while pointing to future work on language modeling integration and BN-independent alignment to handle more complex, multi-writer datasets.

Abstract

This paper serves to introduce the Align, Minimize and Diversify (AMD) method, a Source-Free Unsupervised Domain Adaptation approach for Handwritten Text Recognition (HTR). This framework decouples the adaptation process from the source data, thus not only sidestepping the resource-intensive retraining process but also making it possible to leverage the wealth of pre-trained knowledge encoded in modern Deep Learning architectures. Our method explicitly eliminates the need to revisit the source data during adaptation by incorporating three distinct regularization terms: the Align term, which reduces the feature distribution discrepancy between source and target data, ensuring the transferability of the pre-trained representation; the Minimize term, which encourages the model to make assertive predictions, pushing the outputs towards one-hot-like distributions in order to minimize prediction uncertainty, and finally, the Diversify term, which safeguards against the degeneracy in predictions by promoting varied and distinctive sequences throughout the target data, preventing informational collapse. Experimental results from several benchmarks demonstrated the effectiveness and robustness of AMD, showing it to be competitive and often outperforming DA methods in HTR.

Figures (6)

• Figure 1: Scheme of the proposed Align, Minimize and Diversify (AMD) method. The approach consists of two stages: (i) pre-training the HTR model using a labeled source dataset, and (ii) adapting it to recognize a new target domain using only its images in an unsupervised manner. AMD uses three loss terms to fine-tune the pre-trained source model: (i) the Align term, $\mathcal{L}_a$, to align source and target graphical feature distributions; (ii) the Minimize term, $\mathcal{L}_m$, to guide the frame-wise predictions towards one-hot-like vectors, and (iii) the Diversify term, $\mathcal{L}_d$, to ensure diverse sequences throughout the target data. $w_a$, $w_m$, and $w_d$ are hyper-parameters that control the importance of each term in the loss. The layers shaded in gray in the second stage remain fixed during the adaptation.
• Figure 2: Word samples of the three real corpora used in the experiments. From left to right: IAM, GW and ESPOSALLES.
• Figure 3: Word samples produced by the synthetic generator used in this work.
• Figure 4: Normalized progression of the validation CER metric, the AMD loss and each individual regularization term across epochs for each source-target configuration. In all cases, synthetic data is used as the source data.
• Figure 5: Distribution of CER values across the different writers of IAM after AMD adaptation. The dashed joint line indicates the baseline (no AMD). We used synthetic data as the source data used for pre-training.