Engineering the Law-Machine Learning Translation Problem: Developing Legally Aligned Models

TL;DR

The paper tackles the challenge of aligning ML models with multiple, uncertain legal obligations while preserving predictive performance. It introduces a five-stage interdisciplinary framework that translates legal requirements into operationalizations and evaluation metrics, trains models across diverse operationalization sets, maps trade-offs, and justifies model choices with legal reasoning. A detailed AML case study demonstrates how to derive operationalizations, evaluate proxies for legal obligations, and select a model with auditable legal justification. The approach aims to support compliant, accountable ML deployment in regulated sectors and provides regulators with a structured method to assess due diligence and oversight. Overall, the framework offers a practical path to integrate law and ML design, enabling legally sound and high-performing AI systems.

Abstract

Organizations developing machine learning-based (ML) technologies face the complex challenge of achieving high predictive performance while respecting the law. This intersection between ML and the law creates new complexities. As ML model behavior is inferred from training data, legal obligations cannot be operationalized in source code directly. Rather, legal obligations require "indirect" operationalization. However, choosing context-appropriate operationalizations presents two compounding challenges: (1) laws often permit multiple valid operationalizations for a given legal obligation-each with varying degrees of legal adequacy; and, (2) each operationalization creates unpredictable trade-offs among the different legal obligations and with predictive performance. Evaluating these trade-offs requires metrics (or heuristics), which are in turn difficult to validate against legal obligations. Current methodologies fail to fully address these interwoven challenges as they either focus on legal compliance for traditional software or on ML model development without adequately considering legal complexities. In response, we introduce a five-stage interdisciplinary framework that integrates legal and ML-technical analysis during ML model development. This framework facilitates designing ML models in a legally aligned way and identifying high-performing models that are legally justifiable. Legal reasoning guides choices for operationalizations and evaluation metrics, while ML experts ensure technical feasibility, performance optimization and an accurate interpretation of metric values. This framework bridges the gap between more conceptual analysis of law and ML models' need for deterministic specifications. We illustrate its application using a case study in the context of anti-money laundering.

Figures (1)

• Figure 1: 5-Stage Framework to Design, Evaluate and Select ML Models under Legal Obligations