COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs
As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety.
695c87e66aa73bc11f0914bb
Dasol Choi
claimed_verified
2026-01-06T09:40:31.785Z
66120647cac232c1507e13da
https://cdn-avatars.huggingface.co/v1/production/uploads/66120647cac232c1507e13da/iOBrsyr2mVdVLAFF6pZt6.png
DasolChoi
Dasool
user
Dasool
695c87e66aa73bc11f0914bc
DongGeon Lee
claimed_verified
2026-01-06T09:40:33.640Z
6540fbf9cb7fffd683942b43
https://cdn-avatars.huggingface.co/v1/production/uploads/6540fbf9cb7fffd683942b43/KSB9DPtpF7yv5Dl5vurAH.png
DongGeon Lee
oneonlee
user
oneonlee
695c87e66aa73bc11f0914bd
Brigitta Jesica Kartono
695c87e66aa73bc11f0914be
Helena Berndt
695c87e66aa73bc11f0914bf
Taeyoun Kwon
695c87e66aa73bc11f0914c0
Joonwon Jang
695c87e66aa73bc11f0914c1
Haon Park
695c87e66aa73bc11f0914c2
Hwanjo Yu
695c87e66aa73bc11f0914c3
Minsuk Kahng
COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs
As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety.
COMPASS evaluates large language models' compliance with organizational policies, revealing significant gaps in enforcing prohibitions despite strong performance on legitimate requests.
large language models organizational policies allowlist denylist adversarial robustness policy-critical deployments organizational AI safety