Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Prompt Engineering Approach and a Knowledge Graph based Framework for Tackling Legal Implications of Large Language Model Answers

George Hannah, Rita T. Sousa, Ioannis Dasoulas, Claudia d'Amato

TL;DR

This work provides an empirical analysis on multiple existing LLMs showing the urgency of the problem and proposes a short-term solution consisting in an approach for isolating these legal issues through prompt re-engineering.

Abstract

With the recent surge in popularity of Large Language Models (LLMs), there is the rising risk of users blindly trusting the information in the response, even in cases where the LLM recommends actions that have potential legal implications and this may put the user in danger. We provide an empirical analysis on multiple existing LLMs showing the urgency of the problem. Hence, we propose a short-term solution consisting in an approach for isolating these legal issues through prompt re-engineering. We further analyse the outcomes but also the limitations of the prompt engineering based approach and we highlight the need of additional resources for fully solving the problem We also propose a framework powered by a legal knowledge graph (KG) to generate legal citations for these legal issues, enriching the response of the LLM.

A Prompt Engineering Approach and a Knowledge Graph based Framework for Tackling Legal Implications of Large Language Model Answers

TL;DR

This work provides an empirical analysis on multiple existing LLMs showing the urgency of the problem and proposes a short-term solution consisting in an approach for isolating these legal issues through prompt re-engineering.

Abstract

With the recent surge in popularity of Large Language Models (LLMs), there is the rising risk of users blindly trusting the information in the response, even in cases where the LLM recommends actions that have potential legal implications and this may put the user in danger. We provide an empirical analysis on multiple existing LLMs showing the urgency of the problem. Hence, we propose a short-term solution consisting in an approach for isolating these legal issues through prompt re-engineering. We further analyse the outcomes but also the limitations of the prompt engineering based approach and we highlight the need of additional resources for fully solving the problem We also propose a framework powered by a legal knowledge graph (KG) to generate legal citations for these legal issues, enriching the response of the LLM.

Paper Structure

This paper contains 19 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Analysis and Formulation
  4. Prompt Engineering based Approach
  5. Prompt Template Formalisation
  6. Prompt Engineering Result Samples
  7. Prompt Engineering Limitations
  8. Framework
  9. Process View
  10. Component View
  11. User Interface
  12. Prompt Reformatter
  13. LLM
  14. Knowledge Graph Search
  15. Knowledge Graph Index
  16. ...and 4 more sections

Figures (2)

  • Figure 1: A process-based view of the framework. The dashed lines are used to connect a description of each stage of the process
  • Figure 2: A component-based view of the framework. Yellow refers to functional components, green elements refer to external components or services, and blue elements refer to documents that affect the operation of another component.