Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs & Legal Aid: Understanding Legal Needs Exhibited Through User Queries

Michal Kuk, Jakub Harasta

TL;DR

This study analyzes how users interact with GPT-4-based legal aid rather than evaluating model accuracy, using a Czech public experiment (May 3–July 25, 2023) with 1,252 users and 3,847 queries. It employs zero-shot GPT-4o classification to categorize queries along three dimensions: presence of facts, information versus advice, and user-imposed constraints, revealing that most users seek information (about laws) rather than personalized legal advice and tend to keep the model unconstrained. The findings highlight behavioral patterns such as oversharing, reliance on open-ended prompts, and a spectrum between treating the LLM as a search engine versus a human expert, with implications for AI literacy and safeguards in legal-aid contexts. The work is descriptive and preliminary, calling for more rigorous, demographically informed studies and improved retrieval-augmented strategies to manage risks like hallucinations and misapplication of legal advice, while stressing the ongoing role of LLMs in increasing access to justice.

Abstract

The paper presents a preliminary analysis of an experiment conducted by Frank Bold, a Czech expert group, to explore user interactions with GPT-4 for addressing legal queries. Between May 3, 2023, and July 25, 2023, 1,252 users submitted 3,847 queries. Unlike studies that primarily focus on the accuracy, factuality, or hallucination tendencies of large language models (LLMs), our analysis focuses on the user query dimension of the interaction. Using GPT-4o for zero-shot classification, we categorized queries on (1) whether users provided factual information about their issue (29.95%) or not (70.05%), (2) whether they sought legal information (64.93%) or advice on the course of action (35.07\%), and (3) whether they imposed requirements to shape or control the model's answer (28.57%) or not (71.43%). We provide both quantitative and qualitative insight into user needs and contribute to a better understanding of user engagement with LLMs.

LLMs & Legal Aid: Understanding Legal Needs Exhibited Through User Queries

TL;DR

This study analyzes how users interact with GPT-4-based legal aid rather than evaluating model accuracy, using a Czech public experiment (May 3–July 25, 2023) with 1,252 users and 3,847 queries. It employs zero-shot GPT-4o classification to categorize queries along three dimensions: presence of facts, information versus advice, and user-imposed constraints, revealing that most users seek information (about laws) rather than personalized legal advice and tend to keep the model unconstrained. The findings highlight behavioral patterns such as oversharing, reliance on open-ended prompts, and a spectrum between treating the LLM as a search engine versus a human expert, with implications for AI literacy and safeguards in legal-aid contexts. The work is descriptive and preliminary, calling for more rigorous, demographically informed studies and improved retrieval-augmented strategies to manage risks like hallucinations and misapplication of legal advice, while stressing the ongoing role of LLMs in increasing access to justice.

Abstract

The paper presents a preliminary analysis of an experiment conducted by Frank Bold, a Czech expert group, to explore user interactions with GPT-4 for addressing legal queries. Between May 3, 2023, and July 25, 2023, 1,252 users submitted 3,847 queries. Unlike studies that primarily focus on the accuracy, factuality, or hallucination tendencies of large language models (LLMs), our analysis focuses on the user query dimension of the interaction. Using GPT-4o for zero-shot classification, we categorized queries on (1) whether users provided factual information about their issue (29.95%) or not (70.05%), (2) whether they sought legal information (64.93%) or advice on the course of action (35.07\%), and (3) whether they imposed requirements to shape or control the model's answer (28.57%) or not (71.43%). We provide both quantitative and qualitative insight into user needs and contribute to a better understanding of user engagement with LLMs.
Paper Structure (6 sections, 5 figures, 2 tables)

This paper contains 6 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Experimental set-up
  3. Numbers…
  4. … and needs
  5. Limitations and Discussion
  6. Conclusion and Future Work

Figures (5)

  • Figure 1: Weekly Distribution of Queries
  • Figure 2: Distribution of Queries by Day of Week and Time
  • Figure 3: Query Length Distribution
  • Figure 4: Distribution of Codes (Zero-Shot Classification using GPT-4o; no subsequent evaluation conducted)
  • Figure 5: Proportion of Queries Seeking Personalized Actionable Advice by Treating the Model as a Human Expert (blue) and General Legal Information by Treating the Model as a Search Engine (red)