INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges

TL;DR

The paper presents INACIA, an AI-assisted system that integrates Large Language Models into the Brazilian Federal Court of Accounts (TCU) workflow to accelerate audit-case processing and support adjudication decisions. It combines prompt-driven extraction, retrieval-augmented reasoning, and external knowledge sources to perform basic information extraction, admissibility checks, Periculum in mora and Fumus boni iuris analyses, and generate structured recommendations. A validation dataset (122 cases, 33 variables) and a novel evaluation method show that INACIA’s outputs correlate strongly with human judgments, with GPT-4 achieving 100% accuracy on a key extraction task in validation. While results indicate meaningful speed and productivity gains, the authors acknowledge limitations in coverage and reliability, proposing refinements and highlighting the broader implications of AI-enabled legal processes for public administration globally.

Abstract

This paper introduces INACIA (Instrução Assistida com Inteligência Artificial), a groundbreaking system designed to integrate Large Language Models (LLMs) into the operational framework of Brazilian Federal Court of Accounts (TCU). The system automates various stages of case analysis, including basic information extraction, admissibility examination, Periculum in mora and Fumus boni iuris analyses, and recommendations generation. Through a series of experiments, we demonstrate INACIA's potential in extracting relevant information from case documents, evaluating its legal plausibility, and formulating propositions for judicial decision-making. Utilizing a validation dataset alongside LLMs, our evaluation methodology presents a novel approach to assessing system performance, correlating highly with human judgment. These results underscore INACIA's potential in complex legal task handling while also acknowledging the current limitations. This study discusses possible improvements and the broader implications of applying AI in legal contexts, suggesting that INACIA represents a significant step towards integrating AI in legal systems globally, albeit with cautious optimism grounded in the empirical findings.

Figures (8)

• Figure 1: End-to-end INACIA pipeline (from case documents to initial instruction).
• Figure 2: Examples of files typically found in TCU's case database.
• Figure 3: Pipeline used for Basic Information Extraction.
• Figure 4: Prompt used to extract the allegations and requests in a representação.
• Figure 5: Approach used for Admissibility Examination.