Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach

Yoav Evron, Michal Bar-Asher Siegal, Michael Fire

TL;DR

The paper tackles scalable analysis of illustrations in digitized manuscripts by introducing a modular three-stage pipeline (illustration presence classification, illustration localization, and image captioning). It combines EfficientNet-B0 for page-level filtering, YOLOv11n for precise cropping of illustrations, and the LLaVA vision-language model to generate descriptive captions, with results demonstrated on Vatican Library resources and the Golden Haggadah. The approach achieves end-to-end processing around 0.06 s per page and extracts over 200,000 unique illustrations from more than 3 million pages, accompanied by searchable captions and metadata, enabling semantic retrieval and large-scale visual humanities research. The work highlights significant practical impact for cross-cultural iconographic studies while acknowledging limitations in annotation precision, domain adaptation for captioning, and generalizability beyond the tested corpora, and paves the way for further improvements and broader deployment.

Abstract

The recent Artificial Intelligence (AI) revolution has opened transformative possibilities for the humanities, particularly in unlocking the visual content embedded in historical manuscripts. While digital archives now offer unprecedented access to these materials, the ability to systematically study illustrations at a large scale remains challenging. Our study presents a fast and scalable AI approach for detecting, extracting, and describing illustrations in digitized manuscripts. Focusing on collections like the Vatican Library, our system enables efficient visual analysis across millions of pages. Our pipeline consists of three stages: (1) a fine-tuned image classification model filters out text-only pages; (2) an efficient object detection model identifies and crops illustrations; and (3) a multimodal image captioning model generates concise, human-readable descriptions. These are stored in a searchable database, allowing scholars to retrieve relevant visual materials through keyword queries. By harnessing the power of recent AI advancements, we enable large-scale visual research that was previously impractical, empowering scholars in historical studies, art history, and cultural heritage to explore visual motifs, artistic styles, and cross-cultural influences with new precision and speed. Applying our pipeline to over three million digitized manuscript pages, we automatically identified and extracted more than 200,000 unique illustrations. This scale of processing in under 0.06 seconds per page, dramatically outperforms traditional segmentation techniques in both efficiency and accessibility for visual scholarship. Our work demonstrates how cutting-edge AI tools can profoundly reshape scholarly workflows and open new avenues for multidisciplinary research in the age of digital manuscripts.

Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach

TL;DR

The paper tackles scalable analysis of illustrations in digitized manuscripts by introducing a modular three-stage pipeline (illustration presence classification, illustration localization, and image captioning). It combines EfficientNet-B0 for page-level filtering, YOLOv11n for precise cropping of illustrations, and the LLaVA vision-language model to generate descriptive captions, with results demonstrated on Vatican Library resources and the Golden Haggadah. The approach achieves end-to-end processing around 0.06 s per page and extracts over 200,000 unique illustrations from more than 3 million pages, accompanied by searchable captions and metadata, enabling semantic retrieval and large-scale visual humanities research. The work highlights significant practical impact for cross-cultural iconographic studies while acknowledging limitations in annotation precision, domain adaptation for captioning, and generalizability beyond the tested corpora, and paves the way for further improvements and broader deployment.

Abstract

The recent Artificial Intelligence (AI) revolution has opened transformative possibilities for the humanities, particularly in unlocking the visual content embedded in historical manuscripts. While digital archives now offer unprecedented access to these materials, the ability to systematically study illustrations at a large scale remains challenging. Our study presents a fast and scalable AI approach for detecting, extracting, and describing illustrations in digitized manuscripts. Focusing on collections like the Vatican Library, our system enables efficient visual analysis across millions of pages. Our pipeline consists of three stages: (1) a fine-tuned image classification model filters out text-only pages; (2) an efficient object detection model identifies and crops illustrations; and (3) a multimodal image captioning model generates concise, human-readable descriptions. These are stored in a searchable database, allowing scholars to retrieve relevant visual materials through keyword queries. By harnessing the power of recent AI advancements, we enable large-scale visual research that was previously impractical, empowering scholars in historical studies, art history, and cultural heritage to explore visual motifs, artistic styles, and cross-cultural influences with new precision and speed. Applying our pipeline to over three million digitized manuscript pages, we automatically identified and extracted more than 200,000 unique illustrations. This scale of processing in under 0.06 seconds per page, dramatically outperforms traditional segmentation techniques in both efficiency and accessibility for visual scholarship. Our work demonstrates how cutting-edge AI tools can profoundly reshape scholarly workflows and open new avenues for multidisciplinary research in the age of digital manuscripts.
Paper Structure (26 sections, 5 figures, 1 table)

This paper contains 26 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Digitization
  4. Illustration Detection in Historical Manuscripts
  5. Image Captioning and Vision-Language Models
  6. Methods
  7. Data Acquisition
  8. Illustration Presence Classification
  9. Data Preparation and Labeling
  10. Class Imbalance
  11. Evaluation Metrics
  12. EfficientNet Fine-Tuning
  13. Illustration Localization - Object Detection
  14. Data Preparation and Annotation
  15. Evaluation
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Examples of images with and without illustration. The images on the right contain illustration, while the ones on the left do not
  • Figure 2: Our pipeline for transforming a vast collection of scanned historical document pages into a searchable system for artwork and illustrations.
  • Figure 3: Examples of detected illustrations using our fine-tuned YOLOv11n. Bounding boxes highlight illustration regions.
  • Figure 4: Examples of captions generated by LLaVA for manuscript illustrations.
  • Figure 5: Example search using our prototype interface on the Golden Haggadah.