PASTA: A Scalable Framework for Multi-Policy AI Compliance Evaluation

TL;DR

PASTA tackles the expansion of AI policy requirements by unifying diverse regulations into a common policy-paragraph schema and evaluating descriptive model-card inputs with an LLM-based pairwise engine. It introduces a model-card template, policy normalization, and cost-saving techniques (policy chunking and irrelevancy mapping) and delivers heatmap-based, actionable reports. Expert judgments show strong alignment with human regulators, with Spearman correlations $\rho = 0.626$ for violation and $\rho = 0.761$ for relevance, and stable MAE across cases. Usability testing with AI practitioners indicates the outputs are actionable and decision-ready, suggesting PASTA can democratize governance for individuals and small teams. The work demonstrates a viable, open, modular framework for integrating evolving AI policies into real-world development workflows.

Abstract

AI compliance is becoming increasingly critical as AI systems grow more powerful and pervasive. Yet the rapid expansion of AI policies creates substantial burdens for resource-constrained practitioners lacking policy expertise. Existing approaches typically address one policy at a time, making multi-policy compliance costly. We present PASTA, a scalable compliance tool integrating four innovations: (1) a comprehensive model-card format supporting descriptive inputs across development stages; (2) a policy normalization scheme; (3) an efficient LLM-powered pairwise evaluation engine with cost-saving strategies; and (4) an interface delivering interpretable evaluations via compliance heatmaps and actionable recommendations. Expert evaluation shows PASTA's judgments closely align with human experts ($ρ\geq .626$). The system evaluates five major policies in under two minutes at approximately \$3. A user study (N = 12) confirms practitioners found outputs easy-to-understand and actionable, introducing a novel framework for scalable automated AI governance.

Figures (11)

• Figure 1: Overview of PASTA’s system workflow, from user-provided model card input to structured, multi-policy compliance evaluation output. The system integrates irrelevancy filtering, pairwise comparisons, violation scoring, and aggregation into interpretable outputs such as summaries, tables, and heatmaps.
• Figure 2: Illustration of how raw policy documents are segmented into paragraph-level units, standardized into a table format, and scored for relevancy against model card sections. This process generates irrelevancy maps that filter non-informative comparisons and produce a structured policy dataset for evaluation.
• Figure 3: Illustration of pairwise evaluation: a single model-card section (e.g., "Contact Information") is compared against AIDA policy articles. Multiple articles are batched in one LLM request to reduce redundant calls, but the logic evaluates one section--article pair at a time, yielding one output table row per pair.
• Figure 4: Heatmap interface of evaluation results across all comparison pairs. Each cell encodes violation severity, allowing users to detect patterns and compare compliance gaps across policies and system components.
• Figure 5: Section-wise Evaluation Interface linking compliance issues to specific model card sections. Each row presents the user’s documentation, detected issues, and concrete recommendations for remediation, supporting actionable system improvements. This table also supports policy-specific filtering