Detecting and recognizing characters in Greek papyri with YOLOv8, DeiT and SimCLR

TL;DR

This study tackles automated detection and recognition of Greek letters in papyrus manuscripts, addressing the challenge of isolating individual characters from historical images. The authors propose an ensemble pipeline that uses YOLOv8 for detection and preliminary recognition, then refines predictions with two recognition-focused models: a self-supervised SimCLR-pretrained ResNet-50 and a supervised DeiT transformer. They combine these predictions with Weighted Boxes Fusion and majority voting, leveraging extensive supplementary data from the Oxyrhynchus Papyri and AL-PUB, to achieve competitive results (e.g., mAP of 51.42% for detection and 42.16% for recognition across IoU thresholds 0.5–0.95; 93.2% mAP at IoU 0.5 and 98.6% recall). The work demonstrates the practical potential of AI-assisted transcription for large-scale manuscript analysis and provides publicly released annotations to facilitate further research.

Abstract

Purpose: The capacity to isolate and recognize individual characters from facsimile images of papyrus manuscripts yields rich opportunities for digital analysis. For this reason the `ICDAR 2023 Competition on Detection and Recognition of Greek Letters on Papyri' was held as part of the 17th International Conference on Document Analysis and Recognition. This paper discusses our submission to the competition. Methods: We used an ensemble of YOLOv8 models to detect and classify individual characters and employed two different approaches for refining the character predictions, including a transformer based DeiT approach and a ResNet-50 model trained on a large corpus of unlabelled data using SimCLR, a self-supervised learning method. Results: Our submission won the recognition challenge with a mAP of 42.2%, and was runner-up in the detection challenge with a mean average precision (mAP) of 51.4%. At the more relaxed intersection over union threshold of 0.5, we achieved the highest mean average precision and mean average recall results for both detection and classification. Conclusion: The results demonstrate the potential for these techniques for automated character recognition on historical manuscripts. We ran the prediction pipeline on more than 4,500 images from the Oxyrhynchus Papyri to illustrate the utility of our approach, and we release the results publicly in multiple formats.

Figures (4)

• Figure 1: YOLOv8 results. The boxplots show the five results trained with a different validation partition. The plots on the left show the results on the held-out validation set without using the pseudo-labels from the Oxyrhynchus Papyri. The plots in the center show the improved results on the respective validation sets with the pseudo-labels. The plots on the right show the results on the test dataset. The orange line represents an ensemble of the models at the three resolutions. The horizontal purple line shows the result of the ensemble of all trained models.
• Figure 2: Recognition model results. The plot on the left shows accuracy of SimCLR and DeiT models for recognition of cropped characters on the validation partitions. The plot on the right shows the mAP on the test dataset. The YOLOv8 model ensembles of all resolutions from fig. \ref{['fig:yolov8-results']}c.
• Figure 3: Competition results. The best submission for each metric is shown in yellow. The average recall results for KittyDetection were not available.
• Figure 4: Model output for papyrus in P. Oxy. LVII.3895 (Thucydides iii 23, 94). Courtesy of The Egypt Exploration Society and the Faculty of Classics, University of Oxford.