COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs

TL;DR

COMPASS presents a scalable framework for evaluating organization-specific policy alignment in LLMs by automatically generating policy-aligned queries (base and edge) from allowlists and denylists and judging responses with an LLM. Across eight industry domains and 5,920 verified queries, the study reveals a pronounced asymmetry: models consistently comply with allowed content but struggle to refuse denylisted content, especially under adversarial edge cases. Mitigation strategies such as explicit refusal prompts, few-shot demonstrations, and pre-filtering offer conditional improvements, with pre-filtering delivering the strongest yet imperfect gains (significant denylist accuracy but notable over-refusal of allowed queries). The results highlight a fundamental limitation in current LLM policy reasoning for enterprise safety, while also showing that policy alignment can be learned and generalized through targeted fine-tuning, suggesting a path forward for robust organization-specific safety in production systems.

Abstract

As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety.

Figures (5)

• Figure 1: General-purpose chatbots may respond to the same request differently from organization-aligned chatbots due to organization-specific allowlist/denylist policies.
• Figure 2: Overview of the Compass framework. Given an organization’s allowlist and denylist policies, Compass generates base queries that directly reflect policy intent, as well as edge queries that probe policy boundaries, for example via adversarial transformations. The organization's chatbot responds to these queries, and an LLM judge evaluates each response as aligned or misaligned with the policies.
• Figure 3: Policy alignment as a function of model size for the Gemma-3 and Qwen2.5 families under system prompt-based instantiation. Each line shows one query type (allowed/denied, base/edge). Scaling clearly strengthens compliance on allowlist queries, while denylist robustness remains weak across sizes.
• Figure 4: Comparison of model performance on PAS (%) with and without RAG across query types.
• Figure 5: Failure mode distribution for misaligned Denied-Edge responses. Proprietary models mostly exhibit hybrid failures (refusal followed by compliance), while open-weight models show direct violations.