Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Legal Writing Evaluation of LLMs

Ramon Pires, Roseval Malaquias Junior, Rodrigo Nogueira

TL;DR

The paper introduces oab-bench, a 105-question benchmark derived from the Brazilian Bar Exam to evaluate legal writing with open-ended responses. It provides an automated evaluation pipeline using LLMs as judges, guided by official grading guidelines, and investigates correlations with human scoring by comparing LLM judgments to actual exam judgments. Claude-3-5 Sonnet achieves the best overall performance with an average of $7.93$ out of $10$, and passes all $21$ exams, while frontier models like o1 show reasonable alignment with human scoring in evaluating approved exams. The work also reports on the challenges and costs of automated judging and releases the dataset and code publicly to enable further research in automated evaluation of open-ended tasks in specialized domains.

Abstract

Despite the recent advances in Large Language Models, benchmarks for evaluating legal writing remain scarce due to the inherent complexity of assessing open-ended responses in this domain. One of the key challenges in evaluating language models on domain-specific tasks is finding test datasets that are public, frequently updated, and contain comprehensive evaluation guidelines. The Brazilian Bar Examination meets these requirements. We introduce oab-bench, a benchmark comprising 105 questions across seven areas of law from recent editions of the exam. The benchmark includes comprehensive evaluation guidelines and reference materials used by human examiners to ensure consistent grading. We evaluate the performance of four LLMs on oab-bench, finding that Claude-3.5 Sonnet achieves the best results with an average score of 7.93 out of 10, passing all 21 exams. We also investigated whether LLMs can serve as reliable automated judges for evaluating legal writing. Our experiments show that frontier models like OpenAI's o1 achieve a strong correlation with human scores when evaluating approved exams, suggesting their potential as reliable automated evaluators despite the inherently subjective nature of legal writing assessment. The source code and the benchmark -- containing questions, evaluation guidelines, model-generated responses, and their respective automated evaluations -- are publicly available.

Automatic Legal Writing Evaluation of LLMs

TL;DR

The paper introduces oab-bench, a 105-question benchmark derived from the Brazilian Bar Exam to evaluate legal writing with open-ended responses. It provides an automated evaluation pipeline using LLMs as judges, guided by official grading guidelines, and investigates correlations with human scoring by comparing LLM judgments to actual exam judgments. Claude-3-5 Sonnet achieves the best overall performance with an average of out of , and passes all exams, while frontier models like o1 show reasonable alignment with human scoring in evaluating approved exams. The work also reports on the challenges and costs of automated judging and releases the dataset and code publicly to enable further research in automated evaluation of open-ended tasks in specialized domains.

Abstract

Despite the recent advances in Large Language Models, benchmarks for evaluating legal writing remain scarce due to the inherent complexity of assessing open-ended responses in this domain. One of the key challenges in evaluating language models on domain-specific tasks is finding test datasets that are public, frequently updated, and contain comprehensive evaluation guidelines. The Brazilian Bar Examination meets these requirements. We introduce oab-bench, a benchmark comprising 105 questions across seven areas of law from recent editions of the exam. The benchmark includes comprehensive evaluation guidelines and reference materials used by human examiners to ensure consistent grading. We evaluate the performance of four LLMs on oab-bench, finding that Claude-3.5 Sonnet achieves the best results with an average score of 7.93 out of 10, passing all 21 exams. We also investigated whether LLMs can serve as reliable automated judges for evaluating legal writing. Our experiments show that frontier models like OpenAI's o1 achieve a strong correlation with human scores when evaluating approved exams, suggesting their potential as reliable automated evaluators despite the inherently subjective nature of legal writing assessment. The source code and the benchmark -- containing questions, evaluation guidelines, model-generated responses, and their respective automated evaluations -- are publicly available.
Paper Structure (15 sections, 4 figures, 4 tables)

This paper contains 15 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic evaluation of LLMs on legal domain
  4. Automatic evaluation of open-ended tasks
  5. Methodology
  6. OAB exam
  7. The oab-bench
  8. LLM as a Candidate
  9. LLM as a Judge
  10. Comparison of Judges: Human versus LLM
  11. Results
  12. Evaluating Human Responses
  13. Limitations
  14. Conclusion
  15. Disagreement Analysis in Legal Essays

Figures (4)

  • Figure 1: Example of a commented answer and a score distribution table from the Criminal Law exam (edition 41, question 4). The table shows how the scores are allocated for each item and its parts, with some items accepting multiple valid legal articles as basis.
  • Figure 2: Prompt used to instruct the LLM to act as an examinee for the OAB Exam. The prompt includes the same guidelines given to the candidates in the application of the exam.
  • Figure 3: Prompt template used to instruct the LLM to act as an examiner for the OAB Exam. The prompt explains the analytical grading process and stablishes the format of a final score.
  • Figure 4: Performance of each model across different areas of law.