Running a Data Integration Lab in the Context of the EHRI Project: Challenges, Lessons Learnt and Future Directions

TL;DR

The paper addresses the challenge of dispersed Holocaust archival sources by detailing the EHRI Portal and the EHRI-3 Data Integration Lab, which together lower participation barriers for smaller institutions and enable trans-national data integration. It documents the evolution of data integration tools from XML-centered pipelines to a web-based ETL platform, while showcasing several case studies that reveal technical and social challenges, including format heterogeneity, hierarchical representation, and legal governance. The study contributes a structured methodology, a transparent workflow for data providers, and a blueprint for federated, community-driven data integration in the context of EHRI-ERIC. Its findings have practical significance for GLAM collaborations, suggesting scalable approaches to achieve broader, more sustainable access to Holocaust archival descriptions across multiple national contexts.

Abstract

Historical study of the Holocaust is commonly hampered by the dispersed and fragmented nature of important archival sources relating to this event. The EHRI project set out to mitigate this problem by building a trans-national network of archives, researchers, and digital practitioners, and one of its main outcomes was the creation of the EHRI Portal, a "virtual observatory" that gathers in one centralised platform descriptions of Holocaust-related archival sources from around the world. In order to build the Portal a strong data identification and integration effort was required, culminating in the project's third phase with the creation of the EHRI-3 data integration lab. The focus of the lab was to lower the bar to participation in the EHRI Portal by providing support to institutions in conforming their archival metadata with that required for integration, ultimately opening the process up to smaller institutions (and even so-called "micro-archives") without the necessary resources to undertake this process themselves. In this paper we present our experiences from running the data integration lab and discuss some of the challenges (both of a technical and social nature), how we tried to overcome them, and the overall lessons learnt. We envisage this work as an archetype upon which other practitioners seeking to pursue similar data integration activities can build their own efforts.

Figures (2)

• Figure 1: This diagram represents the general data model used by the EHRI Portal where countries use a custom-based model for representing the EHRI country reports, archives use the ISDIAH standard, archival descriptions are based on the ISAD(G), authority sets follow the ISAAR(CPF), and vocabularies are modelled using SKOS. The relations are modelled as follows: countries can contain an arbitrary number of archives, and archives can hold many archival descriptions. Both archives and archival descriptions can be copied or be the original source of other archives and/or archival descriptions. Finally, archival descriptions might be linked to subjects represented in the vocabularies sets or in the authority sets but only entities pertaining to the authority sets can be stated as the creators of an archival description.
• Figure 2: Diagram of the system implemented to transform JSON files harvested from an API into EAD using the ShExML engine, prior to the standard EHRI Portal data transformation workflow.