Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education

Lorenz Wendlinger, Christian Braun, Abdullah Al Zubaer, Simon Alexander Nonn, Sarah Großkopf, Christofer Fellicious, Michael Granitzer

TL;DR

This study evaluates open-source foundational LLMs for German legal education, focusing on the Gutachtenstil appraisal framework, argument mining, and automated essay scoring. It compares GPT-3.5, Llama 3, Mixtral, and Jina Embeddings across four datasets, employing prompt strategies such as RAG and Chain-of-Thought to probe zero-shot and few-shot performance. Key findings show that while pre-trained LLMs capture German legal background, they struggle with complex tasks like Gutachtenstil integration and full legal opinions, though Retrieval Augmented Generation and few-shot prompting can boost performance in data-rich settings. The results suggest practical utility for simpler tasks and educational prompting, yet underscore limitations in language-specific reasoning, efficiency, and cross-domain transfer, informing future research on task-focused LLM deployment in legal education.

Abstract

We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case.

On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Education

TL;DR

This study evaluates open-source foundational LLMs for German legal education, focusing on the Gutachtenstil appraisal framework, argument mining, and automated essay scoring. It compares GPT-3.5, Llama 3, Mixtral, and Jina Embeddings across four datasets, employing prompt strategies such as RAG and Chain-of-Thought to probe zero-shot and few-shot performance. Key findings show that while pre-trained LLMs capture German legal background, they struggle with complex tasks like Gutachtenstil integration and full legal opinions, though Retrieval Augmented Generation and few-shot prompting can boost performance in data-rich settings. The results suggest practical utility for simpler tasks and educational prompting, yet underscore limitations in language-specific reasoning, efficiency, and cross-domain transfer, informing future research on task-focused LLM deployment in legal education.

Abstract

We show that current open-source foundational LLMs possess instruction capability and German legal background knowledge that is sufficient for some legal analysis in an educational context. However, model capability breaks down in very specific tasks, such as the classification of "Gutachtenstil" appraisal style components, or with complex contexts, such as complete legal opinions. Even with extended context and effective prompting strategies, they cannot match the Bag-of-Words baseline. To combat this, we introduce a Retrieval Augmented Generation based prompt example selection method that substantially improves predictions in high data availability scenarios. We further evaluate the performance of pre-trained LLMs on two standard tasks for argument mining and automated essay scoring and find it to be more adequate. Throughout, pre-trained LLMs improve upon the baseline in scenarios with little or no labeled data with Chain-of-Thought prompting further helping in the zero-shot case.

Paper Structure

This paper contains 25 sections, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Generative Pre-trained Transformer
  5. Llama 3
  6. Mixtral
  7. Jina Embeddings
  8. Prompting Strategies
  9. Results
  10. Datasets
  11. SPWSLE
  12. Graded Strafrecht Hausarbeiten
  13. ASAP AES
  14. CIMT
  15. Evaluation Methodology
  16. ...and 10 more sections

Figures (6)

  • Figure 1: Spearman ranking correlation for the ASAP AES transfer study, mean over 3 folds.
  • Figure 2: Translated system prompt for SPWSLE Gutachtenstil argument mining where $<$EXPLANATION$>$ defines the Gutachtenstil.
  • Figure 3: Translated result prompt for SPWSLE Gutachtenstil argument mining.
  • Figure 4: Chain-of-Thought prompt for SPWSLE Gutachtenstil argument mining
  • Figure 5: Mean proportion of shots with the correct label for a request in SPWSLE Gutachtenstil argument mining. Random selection vs. RAG-based selection via cosine similarity over sentence embeddings.
  • ...and 1 more figures