Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SARCH: Multimodal Search for Archaeological Archives

Nivedita Sinha, Bharati Khanijo, Sanskar Singh, Priyansh Mahant, Ashutosh Roy, Saubhagya Singh Bhadouria, Arpan Jain, Maya Ramanath

TL;DR

The paper addresses the challenge of searching historical archaeological archives that are stored as low-quality scanned PDFs with multi-modal content. It introduces SARCH, an end-to-end system that extracts text, images (classified into maps, photos, layouts, and figures), and tables, enriches them with contextual information, and supports text, image, and table queries via keyword, embedding, and hybrid retrieval. The approach leverages MiniLM for text, CLIP for images, and TAPAS for tables, with context extraction and a reciprocal rank fusion hybrid. Preliminary evaluations on a custom archaeologist-curated benchmark show embedding-based search often performs best, though keyword search remains valuable for archaeology-specific terms, highlighting the importance of context-aware multi-modal retrieval for digital archaeology.

Abstract

In this paper, we describe a multi-modal search system designed to search old archaeological books and reports. This corpus is digitally available as scanned PDFs, but varies widely in the quality of scans. Our pipeline, designed for multi-modal archaeological documents, extracts and indexes text, images (classified into maps, photos, layouts, and others), and tables. We evaluated different retrieval strategies, including keyword-based search, embedding-based models, and a hybrid approach that selects optimal results from both modalities. We report and analyze our preliminary results and discuss future work in this exciting vertical.

SARCH: Multimodal Search for Archaeological Archives

TL;DR

The paper addresses the challenge of searching historical archaeological archives that are stored as low-quality scanned PDFs with multi-modal content. It introduces SARCH, an end-to-end system that extracts text, images (classified into maps, photos, layouts, and figures), and tables, enriches them with contextual information, and supports text, image, and table queries via keyword, embedding, and hybrid retrieval. The approach leverages MiniLM for text, CLIP for images, and TAPAS for tables, with context extraction and a reciprocal rank fusion hybrid. Preliminary evaluations on a custom archaeologist-curated benchmark show embedding-based search often performs best, though keyword search remains valuable for archaeology-specific terms, highlighting the importance of context-aware multi-modal retrieval for digital archaeology.

Abstract

In this paper, we describe a multi-modal search system designed to search old archaeological books and reports. This corpus is digitally available as scanned PDFs, but varies widely in the quality of scans. Our pipeline, designed for multi-modal archaeological documents, extracts and indexes text, images (classified into maps, photos, layouts, and others), and tables. We evaluated different retrieval strategies, including keyword-based search, embedding-based models, and a hybrid approach that selects optimal results from both modalities. We report and analyze our preliminary results and discuss future work in this exciting vertical.

Paper Structure

This paper contains 9 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Offline Processing
  4. Content Extraction
  5. Embeddings Generation and Indexing
  6. Search Architecture
  7. Evaluation
  8. Conclusion and Future Works
  9. GenAI Usage Disclosure

Figures (5)

  • Figure 1: Complex layouts and bad quality scans
  • Figure 2: Offline processing: content extraction, embedding generation and indexing
  • Figure 3: Rotated Picture on page
  • Figure 4: Search Pipelines: 1. The first pipeline takes in the user's query and modality and searches the appropriate vector DB, 2. The second pipeline does a keyword search using the BM25 scoring model, 3. The third pipeline is a hybrid method that combines and reranks results from the previous two searches.
  • Figure 5: Interface of our search system: showing results for queries in different modalities