Label Indeterminacy in AI & Law
Cor Steging, Tadeusz Zbiegień
TL;DR
This paper investigates label indeterminacy in legal machine learning, where outcomes can be shaped by interventions such as settlements or appeals and thus are not fixed ground truth. Using ECtHR Article 6 cases, it treats Grand Chamber decisions as ground truth and Chamber decisions as indeterminate, then contrasts nine label-imputation methods across multiple balanced training sets. The results demonstrate that the choice of imputation strategy materially alters model predictions and evaluation metrics, highlighting a risk of biased or unreliable inferences when indeterminacy is ignored. The work argues for explicit acknowledgment and careful handling of label indeterminacy in AI applications within law, with broad implications for design, training, and evaluation of predictive systems in legal contexts.
Abstract
Machine learning is increasingly used in the legal domain, where it typically operates retrospectively by treating past case outcomes as ground truth. However, legal outcomes are often shaped by human interventions that are not captured in most machine learning approaches. A final decision may result from a settlement, an appeal, or other procedural actions. This creates label indeterminacy: the outcome could have been different if the intervention had or had not taken place. We argue that legal machine learning applications need to account for label indeterminacy. Methods exist that can impute these indeterminate labels, but they are all grounded in unverifiable assumptions. In the context of classifying cases from the European Court of Human Rights, we show that the way that labels are constructed during training can significantly affect model behaviour. We therefore position label indeterminacy as a relevant concern in AI & Law and demonstrate how it can shape model behaviour.