Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Getting in the Door: Streamlining Intake in Civil Legal Services with Large Language Models

Quinten Steenhuis, Hannes Westermann

TL;DR

This paper tackles the bottleneck of civil legal aid intake by integrating formal eligibility rules with large language models in a Docassemble-based housing intake platform. It evaluates eight LLMs, finding that GPT-4-turbo achieves the best overall F1 score around 0.82, with a strong performance in jurisdictions that have more detailed rules. The study demonstrates that LLMs can safely pre-screen applicants, eliciting missing information through follow-ups while reducing false negatives, though errors mostly pertain to follow-up planning and rule interpretation. The work highlights practical implications for accelerating access to justice and lays out future enhancements, including deeper integration, analytics, and cross-provider eligibility matching to further expand reach.

Abstract

Legal intake, the process of finding out if an applicant is eligible for help from a free legal aid program, takes significant time and resources. In part this is because eligibility criteria are nuanced, open-textured, and require frequent revision as grants start and end. In this paper, we investigate the use of large language models (LLMs) to reduce this burden. We describe a digital intake platform that combines logical rules with LLMs to offer eligibility recommendations, and we evaluate the ability of 8 different LLMs to perform this task. We find promising results for this approach to help close the access to justice gap, with the best model reaching an F1 score of .82, while minimizing false negatives.

Getting in the Door: Streamlining Intake in Civil Legal Services with Large Language Models

TL;DR

This paper tackles the bottleneck of civil legal aid intake by integrating formal eligibility rules with large language models in a Docassemble-based housing intake platform. It evaluates eight LLMs, finding that GPT-4-turbo achieves the best overall F1 score around 0.82, with a strong performance in jurisdictions that have more detailed rules. The study demonstrates that LLMs can safely pre-screen applicants, eliciting missing information through follow-ups while reducing false negatives, though errors mostly pertain to follow-up planning and rule interpretation. The work highlights practical implications for accelerating access to justice and lays out future enhancements, including deeper integration, analytics, and cross-provider eligibility matching to further expand reach.

Abstract

Legal intake, the process of finding out if an applicant is eligible for help from a free legal aid program, takes significant time and resources. In part this is because eligibility criteria are nuanced, open-textured, and require frequent revision as grants start and end. In this paper, we investigate the use of large language models (LLMs) to reduce this burden. We describe a digital intake platform that combines logical rules with LLMs to offer eligibility recommendations, and we evaluate the ability of 8 different LLMs to perform this task. We find promising results for this approach to help close the access to justice gap, with the best model reaching an F1 score of .82, while minimizing false negatives.
Paper Structure (30 sections, 2 figures, 5 tables)

This paper contains 30 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Research Questions
  3. Prior Work
  4. AI for access to justice
  5. LLMs for Legal Reasoning
  6. Spotting legal issues and eliciting facts
  7. Proposed framework
  8. Formal eligibility criteria
  9. Program-specific substantive criteria
  10. Screening with LLM assistance
  11. Experimental design
  12. Dataset
  13. D1 - Initial Response Dataset
  14. D2 - Follow-up Response Dataset
  15. Models
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Figure showing progress from problem description, to follow-up question, to a final recommendation to the applicant. The recommendation is qualified (e.g., it says "you probably qualify") and has a button to allow the applicant to learn more about how AI is used in the decision.
  • Figure 2: Confusion Matrices Comparison