Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis

Da Song, Yuheng Huang, Boqi Chen, Tianshuo Cong, Randy Goebel, Lei Ma, Foutse Khomh

TL;DR

The paper addresses the challenge of evaluating implicit regulatory compliance in LLM-driven tool invocation within high-stakes domains. It introduces LogiSafetyGen, a three-stage framework that converts unstructured regulations into deterministic safety constraints, synthesizes compliant execution traces via logic-guided fuzzing, and masks safety steps to test whether LLMs can infer hidden rules from context. Building on this framework, LogiSafetyBench provides 240 human-verified tasks across financial services, tele-healthcare, and smart home IoT to assess both functional success and strict regulatory adherence using dual-Oracle evaluation. Key findings show that while larger models improve functional reasoning, they often neglect safety constraints, underscoring the need for formal safety evaluation tools; the framework offers a rigorous, scalable path to improve regulatory compliance in autonomous agents and to build trust with regulators and users.

Abstract

The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously enforce mandatory safety constraints. To fill this gap, we introduce LogiSafetyGen, a framework that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety-critical traces. Building on this framework, we construct LogiSafetyBench, a benchmark comprising 240 human-verified tasks that require LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models, despite achieving better functional correctness, frequently prioritize task completion over safety, which results in non-compliant behavior.

Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis

TL;DR

The paper addresses the challenge of evaluating implicit regulatory compliance in LLM-driven tool invocation within high-stakes domains. It introduces LogiSafetyGen, a three-stage framework that converts unstructured regulations into deterministic safety constraints, synthesizes compliant execution traces via logic-guided fuzzing, and masks safety steps to test whether LLMs can infer hidden rules from context. Building on this framework, LogiSafetyBench provides 240 human-verified tasks across financial services, tele-healthcare, and smart home IoT to assess both functional success and strict regulatory adherence using dual-Oracle evaluation. Key findings show that while larger models improve functional reasoning, they often neglect safety constraints, underscoring the need for formal safety evaluation tools; the framework offers a rigorous, scalable path to improve regulatory compliance in autonomous agents and to build trust with regulators and users.

Abstract

The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously enforce mandatory safety constraints. To fill this gap, we introduce LogiSafetyGen, a framework that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety-critical traces. Building on this framework, we construct LogiSafetyBench, a benchmark comprising 240 human-verified tasks that require LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models, despite achieving better functional correctness, frequently prioritize task completion over safety, which results in non-compliant behavior.
Paper Structure (15 sections, 1 equation, 3 figures, 2 tables)

This paper contains 15 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview
  5. Automated Safety Oracle Construction
  6. Logic-Guided Trace Generation
  7. Safety Masking and Instruction Synthesis
  8. Benchmark Construction
  9. Scenario Selection
  10. Human Verification Protocol
  11. Evaluation
  12. Experimental Setup
  13. Evaluation of LogiSafetyGen
  14. Results on LogiSafetyBench
  15. Conclusion

Figures (3)

  • Figure 1: The LogiSafetyGen framework. 1 We translate regulations into grounded LTL oracles. 2Logic-Guided Fuzzer synthesizes compliant traces ($\tau^*$). 3 A safety masking pipeline generates natural language instructions ($I$) that force agents to infer implicit constraints.
  • Figure 2: Pass@1 Rates and Risk Distribution. Comparing Goal-Oriented vs. Workflow-Oriented prompts.
  • Figure 3: Failure Mode Analysis (Goal). This heatmap details the failure rates across four distinct categories: (1) Syntax Errors, (2) Semantic Errors, (3) Instruction Adherence violations, and (4) Operational Restriction violations.