Uncovering the Handwritten Text in the Margins: End-to-end Handwritten Text Detection and Recognition
Liang Cheng, Jonas Frankemölle, Adam Axelsson, Ekta Vats
TL;DR
The paper tackles the challenge of digitizing handwritten marginalia in historical documents, a less-explored information source hampered by data scarcity and highly variable handwriting. It introduces an end-to-end pipeline that detects marginalia with R-CNN variants, segments text into lines and words, and recognizes words with an attention-based encoder–decoder (AttentionHTR), leveraging data augmentation and transfer learning. The approach is evaluated on 513 labeled pages from the Uppsala University Library, demonstrating that Faster R-CNN achieves robust marginalia localization (IoU up to $IoU=0.82$) and that AttentionHTR performs well when segmentation is clean, with limitations arising from poor segmentation and unreadable input. The work contributes reproducible code and pretrained models, providing a foundation for expanding marginalia digitization and enabling language-model enhancements and handwriting synthesis in future work.
Abstract
The pressing need for digitization of historical documents has led to a strong interest in designing computerised image processing methods for automatic handwritten text recognition. However, not much attention has been paid on studying the handwritten text written in the margins, i.e. marginalia, that also forms an important source of information. Nevertheless, training an accurate and robust recognition system for marginalia calls for data-efficient approaches due to the unavailability of sufficient amounts of annotated multi-writer texts. Therefore, this work presents an end-to-end framework for automatic detection and recognition of handwritten marginalia, and leverages data augmentation and transfer learning to overcome training data scarcity. The detection phase involves investigation of R-CNN and Faster R-CNN networks. The recognition phase includes an attention-based sequence-to-sequence model, with ResNet feature extraction, bidirectional LSTM-based sequence modeling, and attention-based prediction of marginalia. The effectiveness of the proposed framework has been empirically evaluated on the data from early book collections found in the Uppsala University Library in Sweden. Source code and pre-trained models are available at Github.