Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chinese Labor Law Large Language Model Benchmark

Zixun Lan, Maochun Xu, Yifan Ren, Rui Wu, Jianghui Zhou, Xueyang Cheng, Jianan Ding Ding, Xinheng Wang, Mingmin Chi, Fei Ma

TL;DR

LabourLawBench introduces a comprehensive, labor-law–focused evaluation suite and LabourLawLLM provides a domain-specialized LLM for Chinese labor law. The framework spans 12 tasks (e.g., statute recall, QA, case classification, welfare calculation, NER, case analysis) across 12 subfields, evaluated with a hybrid mix of objective metrics and GPT-based judgments. Empirical results show LabourLawLLM achieving state-of-the-art performance (aggregate 0.68) and consistent gains across task groups and case types, demonstrating the value of subdomain-specific pretraining and task-aligned instruction tuning. This work offers a scalable methodology for building specialized LLMs in other legal subfields, with implications for reliability, interpretability, and wider access to labor-law resources.

Abstract

Recent advances in large language models (LLMs) have led to substantial progress in domain-specific applications, particularly within the legal domain. However, general-purpose models such as GPT-4 often struggle with specialized subdomains that require precise legal knowledge, complex reasoning, and contextual sensitivity. To address these limitations, we present LabourLawLLM, a legal large language model tailored to Chinese labor law. We also introduce LabourLawBench, a comprehensive benchmark covering diverse labor-law tasks, including legal provision citation, knowledge-based question answering, case classification, compensation computation, named entity recognition, and legal case analysis. Our evaluation framework combines objective metrics (e.g., ROUGE-L, accuracy, F1, and soft-F1) with subjective assessment based on GPT-4 scoring. Experiments show that LabourLawLLM consistently outperforms general-purpose and existing legal-specific LLMs across task categories. Beyond labor law, our methodology provides a scalable approach for building specialized LLMs in other legal subfields, improving accuracy, reliability, and societal value of legal AI applications.

Chinese Labor Law Large Language Model Benchmark

TL;DR

LabourLawBench introduces a comprehensive, labor-law–focused evaluation suite and LabourLawLLM provides a domain-specialized LLM for Chinese labor law. The framework spans 12 tasks (e.g., statute recall, QA, case classification, welfare calculation, NER, case analysis) across 12 subfields, evaluated with a hybrid mix of objective metrics and GPT-based judgments. Empirical results show LabourLawLLM achieving state-of-the-art performance (aggregate 0.68) and consistent gains across task groups and case types, demonstrating the value of subdomain-specific pretraining and task-aligned instruction tuning. This work offers a scalable methodology for building specialized LLMs in other legal subfields, with implications for reliability, interpretability, and wider access to labor-law resources.

Abstract

Recent advances in large language models (LLMs) have led to substantial progress in domain-specific applications, particularly within the legal domain. However, general-purpose models such as GPT-4 often struggle with specialized subdomains that require precise legal knowledge, complex reasoning, and contextual sensitivity. To address these limitations, we present LabourLawLLM, a legal large language model tailored to Chinese labor law. We also introduce LabourLawBench, a comprehensive benchmark covering diverse labor-law tasks, including legal provision citation, knowledge-based question answering, case classification, compensation computation, named entity recognition, and legal case analysis. Our evaluation framework combines objective metrics (e.g., ROUGE-L, accuracy, F1, and soft-F1) with subjective assessment based on GPT-4 scoring. Experiments show that LabourLawLLM consistently outperforms general-purpose and existing legal-specific LLMs across task categories. Beyond labor law, our methodology provides a scalable approach for building specialized LLMs in other legal subfields, improving accuracy, reliability, and societal value of legal AI applications.
Paper Structure (29 sections, 13 equations, 3 figures, 18 tables)

This paper contains 29 sections, 13 equations, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chinese Legal Domain-Specific Language Models
  4. Evaluation Benchmarks for Chinese Legal LLMs
  5. Motivation: Toward a Labor Law Specific Evaluation Framework
  6. Data
  7. Statute Recitation Task
  8. Knowledge Q&A (Single Choice)
  9. Knowledge Q&A (Multiple Choice)
  10. Case-type Prediction
  11. Welfare Compensation Prediction
  12. Named Entity Recognition (NER)
  13. Claim Mining
  14. Disputed Issue Mining
  15. Statute Prediction (Based on Facts)
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Performance of baselines across tasks.
  • Figure 2: Performance of baselines across case types.
  • Figure 3: The Performance of different models.