Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark

Yu Wu, Ke Shu, Jonas Fischer, Lidia Pivovarova, David Rosson, Eetu Mäkelä, Mikko Tolonen

TL;DR

This work defines and benchmarks a novel multimodal task for detecting and extracting Latin fragments in 18th‑century historical books, using a ground-truth dataset of 724 ECCO pages with 12 Latin usage categories. It proposes a unified, model-agnostic prompt pipeline that operates on image, text, or both, and rigorously evaluates contemporary LLMs and a baseline linguistic detector under OCR noise through OCR post-correction and fuzzy token matching. Key findings show reliable Latin detection is achievable in zero-shot settings, with open-source LLMs reaching or surpassing baseline performance, particularly when multimodal input is leveraged; however, category-level analyses reveal limited semantic understanding and strong dependence on superficial textual cues. The study delivers a practical Latin detection pipeline, provides a valuable dataset for further research, and points to future work on semantic grounding, cross-domain generalization, and extension to additional historical languages.

Abstract

This paper presents a novel task of extracting Latin fragments from mixed-language historical documents with varied layouts. We benchmark and evaluate the performance of large foundation models against a multimodal dataset of 724 annotated pages. The results demonstrate that reliable Latin detection with contemporary models is achievable. Our study provides the first comprehensive analysis of these models' capabilities and limits for this task.

Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark

TL;DR

This work defines and benchmarks a novel multimodal task for detecting and extracting Latin fragments in 18th‑century historical books, using a ground-truth dataset of 724 ECCO pages with 12 Latin usage categories. It proposes a unified, model-agnostic prompt pipeline that operates on image, text, or both, and rigorously evaluates contemporary LLMs and a baseline linguistic detector under OCR noise through OCR post-correction and fuzzy token matching. Key findings show reliable Latin detection is achievable in zero-shot settings, with open-source LLMs reaching or surpassing baseline performance, particularly when multimodal input is leveraged; however, category-level analyses reveal limited semantic understanding and strong dependence on superficial textual cues. The study delivers a practical Latin detection pipeline, provides a valuable dataset for further research, and points to future work on semantic grounding, cross-domain generalization, and extension to additional historical languages.

Abstract

This paper presents a novel task of extracting Latin fragments from mixed-language historical documents with varied layouts. We benchmark and evaluate the performance of large foundation models against a multimodal dataset of 724 annotated pages. The results demonstrate that reliable Latin detection with contemporary models is achievable. Our study provides the first comprehensive analysis of these models' capabilities and limits for this task.
Paper Structure (56 sections, 10 figures, 4 tables)

This paper contains 56 sections, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Related work
  4. Latin in NLP
  5. Code-mixed Language Detection
  6. LLMs for Historical Documents
  7. Dataset
  8. Sampling and Annotations
  9. Dataset Characteristics
  10. Evaluation Setting
  11. OCR Post-Correction and Normalization
  12. Metrics
  13. Latin Extraction Pipeline with LLMs
  14. Unified Prompting Strategy
  15. Structured Output and Postprocessing
  16. ...and 41 more sections

Figures (10)

  • Figure 1: An example of an annotated Latin fragment and surrounding context.
  • Figure 2: Macro token recall statistics on different category labels for 5 top-performing models. Qwen2.5 models are all with 32B parameters. The number of page instances with each label tagged is shown in parentheses. Values in the legend indicate each model’s token precision from Table \ref{['tab:results']} to provide complementary performance context.
  • Figure 3: Prompt modification experiment results.
  • Figure 4: An example page with Latin fragments.
  • Figure 5: An example page with Latin fragments.
  • ...and 5 more figures