Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language

TL;DR

This work tackles the challenge of evaluating semantic search in domain-specific German within the process industry, a classic low-resource setting. It introduces an end-to-end annotation pipeline that combines an ensemble of weak text encoders for document indexing with GPT-4o-based query generation and re-ranking, aiming to align automated relevance scores with human judgments. Across seven plant shift-book datasets, the combined ensemble + LLM approach substantially improves inter-coder agreement and ranking metrics compared to single-model baselines, demonstrating a scalable path to creating evaluation data in specialized domains. The methodology and findings offer a practical route to adapt semantic search evaluation in other low-resource languages and domains, balancing automation with targeted human oversight.

Abstract

Domain-specific languages that use a lot of specific terminology often fall into the category of low-resource languages. Collecting test datasets in a narrow domain is time-consuming and requires skilled human resources with domain knowledge and training for the annotation task. This study addresses the challenge of automated collecting test datasets to evaluate semantic search in low-resource domain-specific German language of the process industry. Our approach proposes an end-to-end annotation pipeline for automated query generation to the score reassessment of query-document pairs. To overcome the lack of text encoders trained in the German chemistry domain, we explore a principle of an ensemble of "weak" text encoders trained on common knowledge datasets. We combine individual relevance scores from diverse models to retrieve document candidates and relevance scores generated by an LLM, aiming to achieve consensus on query-document alignment. Evaluation results demonstrate that the ensemble method significantly improves alignment with human-assigned relevance scores, outperforming individual models in both inter-coder agreement and accuracy metrics. These findings suggest that ensemble learning can effectively adapt semantic search systems for specialized, low-resource languages, offering a practical solution to resource limitations in domain-specific contexts.

Figures (4)

• Figure 1: An example of a mocked text log from a shift book in the German language. The logs contain a log of domain-specific terms, which require domain knowledge in the area and know specifics of the production process
• Figure 2: A proposed methodology with ensembles of (L)LM encoders used to retrieve the most relevant documents, i.e., text logs of a shift book, and with an LLM to adjust the relevance score for the document re-ranking.
• Figure 3: The distribution of the relevance scores produced by an ensemble of encoders and GPT-4o. While the ensemble assigns 1 relevance score, GPT-4o leans towards the score of 3. The proposed combined approach balances out these model tendencies.
• Figure 4: The confusion matrices of the annotated vs. automated relevance scores for four methods: an ensemble of encoders, GPT-4o with vague examples, GPT-4o with specific examples (SE), and combined ensemble + GPT-4o-SE. The combined approach allocates most of the results on the matrix diagonal, whereas its components separately lean towards one score or another.