Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LeKUBE: A Legal Knowledge Update BEnchmark

Changyue Wang, Weihang Su, Hu Yiran, Qingyao Ai, Yueyue Wu, Cheng Luo, Yiqun Liu, Min Zhang, Shaoping Ma

TL;DR

A comprehensive evaluation of state-of-the-art knowledge update methods for legal LLMs reveals a notable gap between existing knowledge update methods and the unique needs of the legal domain, emphasizing the need for further research and development of knowledge update mechanisms tailored for legal LLMs.

Abstract

Recent advances in Large Language Models (LLMs) have significantly shaped the applications of AI in multiple fields, including the studies of legal intelligence. Trained on extensive legal texts, including statutes and legal documents, the legal LLMs can capture important legal knowledge/concepts effectively and provide important support for downstream legal applications such as legal consultancy. Yet, the dynamic nature of legal statutes and interpretations also poses new challenges to the use of LLMs in legal applications. Particularly, how to update the legal knowledge of LLMs effectively and efficiently has become an important research problem in practice. Existing benchmarks for evaluating knowledge update methods are mostly designed for the open domain and cannot address the specific challenges of the legal domain, such as the nuanced application of new legal knowledge, the complexity and lengthiness of legal regulations, and the intricate nature of legal reasoning. To address this gap, we introduce the Legal Knowledge Update BEnchmark, i.e. LeKUBE, which evaluates knowledge update methods for legal LLMs across five dimensions. Specifically, we categorize the needs of knowledge updates in the legal domain with the help of legal professionals, and then hire annotators from law schools to create synthetic updates to the Chinese Criminal and Civil Code as well as sets of questions of which the answers would change after the updates. Through a comprehensive evaluation of state-of-the-art knowledge update methods, we reveal a notable gap between existing knowledge update methods and the unique needs of the legal domain, emphasizing the need for further research and development of knowledge update mechanisms tailored for legal LLMs.

LeKUBE: A Legal Knowledge Update BEnchmark

TL;DR

A comprehensive evaluation of state-of-the-art knowledge update methods for legal LLMs reveals a notable gap between existing knowledge update methods and the unique needs of the legal domain, emphasizing the need for further research and development of knowledge update mechanisms tailored for legal LLMs.

Abstract

Recent advances in Large Language Models (LLMs) have significantly shaped the applications of AI in multiple fields, including the studies of legal intelligence. Trained on extensive legal texts, including statutes and legal documents, the legal LLMs can capture important legal knowledge/concepts effectively and provide important support for downstream legal applications such as legal consultancy. Yet, the dynamic nature of legal statutes and interpretations also poses new challenges to the use of LLMs in legal applications. Particularly, how to update the legal knowledge of LLMs effectively and efficiently has become an important research problem in practice. Existing benchmarks for evaluating knowledge update methods are mostly designed for the open domain and cannot address the specific challenges of the legal domain, such as the nuanced application of new legal knowledge, the complexity and lengthiness of legal regulations, and the intricate nature of legal reasoning. To address this gap, we introduce the Legal Knowledge Update BEnchmark, i.e. LeKUBE, which evaluates knowledge update methods for legal LLMs across five dimensions. Specifically, we categorize the needs of knowledge updates in the legal domain with the help of legal professionals, and then hire annotators from law schools to create synthetic updates to the Chinese Criminal and Civil Code as well as sets of questions of which the answers would change after the updates. Through a comprehensive evaluation of state-of-the-art knowledge update methods, we reveal a notable gap between existing knowledge update methods and the unique needs of the legal domain, emphasizing the need for further research and development of knowledge update mechanisms tailored for legal LLMs.
Paper Structure (37 sections, 1 equation, 3 figures, 7 tables)

This paper contains 37 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Update Methods for LLMs
  4. Evaluation of Knowledge Updating
  5. Existing Benchmarks
  6. Challenges of Legal Knowledge Update
  7. Evaluation Dimensions of LeKUBE
  8. Accuracy
  9. Generality
  10. Locality
  11. Scalability
  12. Retainability
  13. Dataset Construction
  14. Data Source
  15. Annotation Recruitment and Payment
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Flowchart illustrating our scalability evaluation by comparing two methodologies. In the blue section, individual LLMs are updated based on knowledge from specific chapters, and their performance is evaluated by average question accuracy. In the red section, a single LLM is updated with knowledge from all chapters that need to be updated in LeKUBE.
  • Figure 2: Flowchart illustrating our retainability evaluation by comparing two phases. The first one is the situation where the criminal law is updated while the civil code is not. And the second one is the situation where the two laws are both updated.
  • Figure 3: Evaluation of Locality. The calculated method of score is described in \ref{['cha:local']}. The higher the score is, the better locality the method has.