Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Prompt Optimisation for Legal Text Classification with Proxy Prompt Evaluator

Hyunji Lee, Kevin Chenhao Li, Matthias Grabmair, Shanshan Xu

TL;DR

This work tackles efficient prompt optimization for legal text classification, focusing on unfair ToS clause detection. It marries Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to explore the prompt space under constrained compute, while updating prompts via textual gradients. The approach is validated on the CLAUDETTE dataset, showing that MCTS with Proxy PromptEval achieves competitive accuracy with substantial cost reductions, approaching the performance of fully supervised baselines without extensive retraining. The contribution advances practical, scalable prompt optimization for high-stakes legal NLP by enabling efficient search and evaluation with a lightweight proxy scorer.

Abstract

Prompt optimization aims to systematically refine prompts to enhance a language model's performance on specific tasks. Fairness detection in Terms of Service (ToS) clauses is a challenging legal NLP task that demands carefully crafted prompts to ensure reliable results. However, existing prompt optimization methods are often computationally expensive due to inefficient search strategies and costly prompt candidate scoring. In this paper, we propose a framework that combines Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to more effectively explore the prompt space while reducing evaluation costs. Experiments demonstrate that our approach achieves higher classification accuracy and efficiency than baseline methods under a constrained computation budget.

Efficient Prompt Optimisation for Legal Text Classification with Proxy Prompt Evaluator

TL;DR

This work tackles efficient prompt optimization for legal text classification, focusing on unfair ToS clause detection. It marries Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to explore the prompt space under constrained compute, while updating prompts via textual gradients. The approach is validated on the CLAUDETTE dataset, showing that MCTS with Proxy PromptEval achieves competitive accuracy with substantial cost reductions, approaching the performance of fully supervised baselines without extensive retraining. The contribution advances practical, scalable prompt optimization for high-stakes legal NLP by enabling efficient search and evaluation with a lightweight proxy scorer.

Abstract

Prompt optimization aims to systematically refine prompts to enhance a language model's performance on specific tasks. Fairness detection in Terms of Service (ToS) clauses is a challenging legal NLP task that demands carefully crafted prompts to ensure reliable results. However, existing prompt optimization methods are often computationally expensive due to inefficient search strategies and costly prompt candidate scoring. In this paper, we propose a framework that combines Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to more effectively explore the prompt space while reducing evaluation costs. Experiments demonstrate that our approach achieves higher classification accuracy and efficiency than baseline methods under a constrained computation budget.

Paper Structure

This paper contains 30 sections, 5 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unfair ToS Clause Detection
  4. Prompt Optimization
  5. Prompt Update
  6. Resampling-based
  7. Implicit reflection-based
  8. Explicit reflection-based
  9. Search Strategy
  10. Greedy search
  11. Beam search
  12. Tree-based search
  13. Prompt Evaluation
  14. Dataset
  15. Methodology
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Our framework with scoring on the score set or alternative scoring with the trained PromptEval-based correctness classifier.
  • Figure 2: Examples from the CLAUDETTE dataset.
  • Figure 3: Prompt update with textual gradients. The global context c is "You are a prompt optimizer for legal documents. The task is to classify clauses of Terms of Service documents according to the given prompt.".
  • Figure 4: Final prompts found with our approach.