Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting

Christian Braun, Alexander Lilienbeck, Daniel Mentjukov

TL;DR

This work investigates how explicit input structure and prompt engineering affect legal QA performance for two GPT-4 series models on a CUAD excerpt. It systematically compares five input modalities (from well-structured TXT to Markdown) and two prompting strategies (user-centric vs system-centric) to assess exact-match accuracy on a bipartite Yes/No plus verbatim clause task. The findings show GPT-4o is robust to input structure yet underperforms GPT-4.1 overall, while GPT-4.1 benefits substantially from well-structured inputs and, especially, from system prompts that acknowledge structure—with Markdown achieving the highest accuracy (~79%). The study underscores that input formatting and prompt design remain critical in high-stakes legal AI, guiding data preparation, model selection, and governance, and it emphasizes the continued need for human-in-the-loop due to imperfect reliability even under optimized conditions.

Abstract

Legal contracts possess an inherent, semantically vital structure (e.g., sections, clauses) that is crucial for human comprehension but whose impact on LLM processing remains under-explored. This paper investigates the effects of explicit input text structure and prompt engineering on the performance of GPT-4o and GPT-4.1 on a legal question-answering task using an excerpt of the CUAD. We compare model exact-match accuracy across various input formats: well-structured plain-text (human-generated from CUAD), plain-text cleaned of line breaks, extracted plain-text from Azure OCR, plain-text extracted by GPT-4o Vision, and extracted (and interpreted) Markdown (MD) from GPT-4o Vision. To give an indication of the impact of possible prompt engineering, we assess the impact of shifting task instructions to the system prompt and explicitly informing the model about the structured nature of the input. Our findings reveal that GPT-4o demonstrates considerable robustness to variations in input structure, but lacks in overall performance. Conversely, GPT-4.1's performance is markedly sensitive; poorly structured inputs yield suboptimal results (but identical with GPT-4o), while well-structured formats (original CUAD text, GPT-4o Vision text and GPT-4o MD) improve exact-match accuracy by ~20 percentage points. Optimizing the system prompt to include task details and an advisory about structured input further elevates GPT-4.1's accuracy by an additional ~10-13 percentage points, with Markdown ultimately achieving the highest performance under these conditions (79 percentage points overall exact-match accuracy). This research empirically demonstrates that while newer models exhibit greater resilience, careful input structuring and strategic prompt design remain critical for optimizing the performance of LLMs, and can significantly affect outcomes in high-stakes legal applications.

The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting

TL;DR

This work investigates how explicit input structure and prompt engineering affect legal QA performance for two GPT-4 series models on a CUAD excerpt. It systematically compares five input modalities (from well-structured TXT to Markdown) and two prompting strategies (user-centric vs system-centric) to assess exact-match accuracy on a bipartite Yes/No plus verbatim clause task. The findings show GPT-4o is robust to input structure yet underperforms GPT-4.1 overall, while GPT-4.1 benefits substantially from well-structured inputs and, especially, from system prompts that acknowledge structure—with Markdown achieving the highest accuracy (~79%). The study underscores that input formatting and prompt design remain critical in high-stakes legal AI, guiding data preparation, model selection, and governance, and it emphasizes the continued need for human-in-the-loop due to imperfect reliability even under optimized conditions.

Abstract

Legal contracts possess an inherent, semantically vital structure (e.g., sections, clauses) that is crucial for human comprehension but whose impact on LLM processing remains under-explored. This paper investigates the effects of explicit input text structure and prompt engineering on the performance of GPT-4o and GPT-4.1 on a legal question-answering task using an excerpt of the CUAD. We compare model exact-match accuracy across various input formats: well-structured plain-text (human-generated from CUAD), plain-text cleaned of line breaks, extracted plain-text from Azure OCR, plain-text extracted by GPT-4o Vision, and extracted (and interpreted) Markdown (MD) from GPT-4o Vision. To give an indication of the impact of possible prompt engineering, we assess the impact of shifting task instructions to the system prompt and explicitly informing the model about the structured nature of the input. Our findings reveal that GPT-4o demonstrates considerable robustness to variations in input structure, but lacks in overall performance. Conversely, GPT-4.1's performance is markedly sensitive; poorly structured inputs yield suboptimal results (but identical with GPT-4o), while well-structured formats (original CUAD text, GPT-4o Vision text and GPT-4o MD) improve exact-match accuracy by ~20 percentage points. Optimizing the system prompt to include task details and an advisory about structured input further elevates GPT-4.1's accuracy by an additional ~10-13 percentage points, with Markdown ultimately achieving the highest performance under these conditions (79 percentage points overall exact-match accuracy). This research empirically demonstrates that while newer models exhibit greater resilience, careful input structuring and strategic prompt design remain critical for optimizing the performance of LLMs, and can significantly affect outcomes in high-stakes legal applications.
Paper Structure (30 sections, 4 figures, 2 tables)

This paper contains 30 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Experiments
  4. Experimental Setup
  5. Dataset and Task Definition
  6. Language Models
  7. Input Text Preparation and Modalities
  8. Prompting Strategies
  9. Strategy 1:
  10. Strategy 2:
  11. Evaluation Metrics
  12. Results
  13. Impact of Input Text Structure (Hypothesis H1)
  14. Impact of Prompt Engineering and Structure Awareness (Hypothesis H2)
  15. Discussion
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Visual comparison of file formats of the same contract section.
  • Figure 2: Results after answering the questions and extracting the relevant clauses for answering (excerpt).
  • Figure 3: Top 20 questions distribution from the used excerpt of the CUAD dataset.
  • Figure 4: Building ground truth from CUAD dataset