Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seventeenth-Century Spanish American Notary Records for Fine-Tuning Spanish Large Language Models

Shraboni Sarker, Ahmad Tamim Hamad, Hulayyil Alshammari, Viviana Grieco, Praveen Rao

TL;DR

This work introduces SANRlite, a curated subset of the 17th-century Spanish American Notary Records designed for fine-tuning Spanish LLMs. By annotating 162 pages with 900+ sentences, 33 class labels, 154 extended labels, Wikidata metadata, and 768-dim sentence embeddings, the authors enable targeted downstream tasks such as sentence classification and masked language modeling. Fine-tuning BETO and M-BERT variants on SANRlite yielded superior performance compared to pre-trained Spanish models and ChatGPT prompts, highlighting the utility of domain-specific historical corpora for NLP in low-resource, archaic-script contexts. The resource, methodology, and empirical results collectively advance historical text analysis and information retrieval, with future work exploring multimodal embeddings (e.g., CLIP) to integrate textual and image data.

Abstract

Large language models have gained tremendous popularity in domains such as e-commerce, finance, healthcare, and education. Fine-tuning is a common approach to customize an LLM on a domain-specific dataset for a desired downstream task. In this paper, we present a valuable resource for fine-tuning LLMs developed for the Spanish language to perform a variety of tasks such as classification, masked language modeling, clustering, and others. Our resource is a collection of handwritten notary records from the seventeenth century obtained from the National Archives of Argentina. This collection contains a combination of original images and transcribed text (and metadata) of 160+ pages that were handwritten by two notaries, namely, Estenban Agreda de Vergara and Nicolas de Valdivia y Brisuela nearly 400 years ago. Through empirical evaluation, we demonstrate that our collection can be used to fine-tune Spanish LLMs for tasks such as classification and masked language modeling, and can outperform pre-trained Spanish models and ChatGPT-3.5/ChatGPT-4o. Our resource will be an invaluable resource for historical text analysis and is publicly available on GitHub.

Seventeenth-Century Spanish American Notary Records for Fine-Tuning Spanish Large Language Models

TL;DR

This work introduces SANRlite, a curated subset of the 17th-century Spanish American Notary Records designed for fine-tuning Spanish LLMs. By annotating 162 pages with 900+ sentences, 33 class labels, 154 extended labels, Wikidata metadata, and 768-dim sentence embeddings, the authors enable targeted downstream tasks such as sentence classification and masked language modeling. Fine-tuning BETO and M-BERT variants on SANRlite yielded superior performance compared to pre-trained Spanish models and ChatGPT prompts, highlighting the utility of domain-specific historical corpora for NLP in low-resource, archaic-script contexts. The resource, methodology, and empirical results collectively advance historical text analysis and information retrieval, with future work exploring multimodal embeddings (e.g., CLIP) to integrate textual and image data.

Abstract

Large language models have gained tremendous popularity in domains such as e-commerce, finance, healthcare, and education. Fine-tuning is a common approach to customize an LLM on a domain-specific dataset for a desired downstream task. In this paper, we present a valuable resource for fine-tuning LLMs developed for the Spanish language to perform a variety of tasks such as classification, masked language modeling, clustering, and others. Our resource is a collection of handwritten notary records from the seventeenth century obtained from the National Archives of Argentina. This collection contains a combination of original images and transcribed text (and metadata) of 160+ pages that were handwritten by two notaries, namely, Estenban Agreda de Vergara and Nicolas de Valdivia y Brisuela nearly 400 years ago. Through empirical evaluation, we demonstrate that our collection can be used to fine-tune Spanish LLMs for tasks such as classification and masked language modeling, and can outperform pre-trained Spanish models and ChatGPT-3.5/ChatGPT-4o. Our resource will be an invaluable resource for historical text analysis and is publicly available on GitHub.
Paper Structure (9 sections, 4 figures, 2 tables)

This paper contains 9 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Description of the Resource
  4. Experiments
  5. Implementation, Setup & Evaluation Metrics
  6. Model Fine-Tuning
  7. Multi-Label Classification: Fine-Tuned LLMs Versus Pre-Trained LLMs
  8. Masked Language Modeling: Fined-Tuned LLMs Versus Pre-Trained LLMs
  9. Conclusion

Figures (4)

  • Figure 1: An example image in SANRlite
  • Figure 2: An example of a notary record in SANRlite and its annotation
  • Figure 3: JSON metadata for a sentence in a notary record
  • Figure 4: Steps for constructing SANRlite and fine tuning BERT-based models for classification and mask language modeling