LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text

TL;DR

This work tackles the lack of standardized evaluation of linguistic quality in LLM-generated legal text by introducing LegalEval-Q, a regression-based framework that assesses five quality dimensions, a dedicated legal-question dataset, and a multi-model analysis of 49 LLMs. It combines an Evaluation Model with a Multi-source Regression Model to produce three outputs (Comment, Conclusion, Score) and defines AdjScore to balance mean performance with stability. Key findings include a plateau in legal-text quality around 7B parameters, no statistically significant gains from quantization or extended context within studied ranges, and consistent advantages for reasoning-oriented models, with Qwen3-series offering favorable cost–performance trade-offs. The framework enables standardized, domain-specific benchmarking and provides practical guidance for model selection in legal tasks, along with public release of code and models to support reproducibility and further research.

Abstract

As large language models (LLMs) are increasingly used in legal applications, current evaluation benchmarks tend to focus mainly on factual accuracy while largely neglecting important linguistic quality aspects such as clarity, coherence, and terminology. To address this gap, we propose three steps: First, we develop a regression model to evaluate the quality of legal texts based on clarity, coherence, and terminology. Second, we create a specialized set of legal questions. Third, we analyze 49 LLMs using this evaluation framework. Our analysis identifies three key findings: First, model quality levels off at 14 billion parameters, with only a marginal improvement of $2.7\%$ noted at 72 billion parameters. Second, engineering choices such as quantization and context length have a negligible impact, as indicated by statistical significance thresholds above 0.016. Third, reasoning models consistently outperform base architectures. A significant outcome of our research is the release of a ranking list and Pareto analysis, which highlight the Qwen3 series as the optimal choice for cost-performance tradeoffs. This work not only establishes standardized evaluation protocols for legal LLMs but also uncovers fundamental limitations in current training data refinement approaches. Code and models are available at: https://github.com/lyxx3rd/LegalEval-Q.

Figures (13)

• Figure 1: Illustration of how evaluation criteria are mapped into comments. The figure serves as a demonstration of the mapping process rather than a complete input–output case. The example is presented in the original Chinese text together with its English translation. Full examples with complete input–output pairs are provided in the appendix B.
• Figure 2: Distribution of model scores with probability density, illustrating central tendency and variance in legal text quality evaluation.
• Figure 3: End-to-end data flow of the evaluation framework, showing how raw inputs are processed through comment generation, conclusion synthesis, and final scoring modules. Abbreviation: Classification Token (CLS), representing sequence-level embeddings extracted from transformer layers.
• Figure 4: Scaling laws of model capabilities: legal text quality (blue) vs. mathematical reasoning (red) across 0.5B–72B models.
• Figure 5: Performance Comparison Between Reasoning Models and Base Models: Evaluating Score (Mean) improvement of reasoning models compared to their base counterparts.