LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models

TL;DR

LalaEval addresses the need for standardized human evaluation of domain-specific LLMs by proposing an end-to-end framework that defines domain scope, evaluation criteria, domain-tailored benchmarks, rubrics, and analysis protocols. The methodology is demonstrated in the logistics domain, where a hierarchical domain specification guides subdomain prioritization, and a structured benchmark and rubric system enables rigorous, multi-dimensional assessment with dispute and stability analyses. Results from zero-shot tests show how domain-specific evaluations can reveal strengths and gaps across general and domain capabilities, guiding model selection and development for practical, domain-aligned use. The framework is designed to be generalizable across industries, offering a standardized, pragmatic approach to evaluating the real-world utility of domain-specific LLMs.

Abstract

This paper introduces LalaEval, a holistic framework designed for the human evaluation of domain-specific large language models (LLMs). LalaEval proposes a comprehensive suite of end-to-end protocols that cover five main components including domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, and thorough analysis and interpretation of evaluation outcomes. This initiative aims to fill a crucial research gap by providing a systematic methodology for conducting standardized human evaluations within specific domains, a practice that, despite its widespread application, lacks substantial coverage in the literature and human evaluation are often criticized to be less reliable due to subjective factors, so standardized procedures adapted to the nuanced requirements of specific domains or even individual organizations are in great need. Furthermore, the paper demonstrates the framework's application within the logistics industry, presenting domain-specific evaluation benchmarks, datasets, and a comparative analysis of LLMs for the logistics domain use, highlighting the framework's capacity to elucidate performance differences and guide model selection and development for domain-specific LLMs. Through real-world deployment, the paper underscores the framework's effectiveness in advancing the field of domain-specific LLM evaluation, thereby contributing significantly to the ongoing discussion on LLMs' practical utility and performance in domain-specific applications.

Figures (2)

• Figure 1: The hierarchical domain specification in the logistics industry
• Figure 2: The overall deployment diagram of LalaEval