Lattice: Generative Guardrails for Conversational Agents

TL;DR

Lattice addresses the brittleness of static guardrails in deployed conversational AI by introducing a two-stage framework: construction, which builds an initial guardrail set from labeled data through iterative simulation, evaluation, and optimization, and continuous improvement, which autonomously adapts deployed guardrails via risk assessment, adversarial case expansion, and consolidation using prompted LLMs. The approach achieves strong holdout performance on ProsocialDialog, with $F_1$ reaching $0.91$ and a compact final guardrail set (6 rules) from 100 labeled examples, outperforming static baselines by up to $43$pp. In cross-domain settings, the continuous-improvement loop yields further gains (up to $+7$pp in $F_1$) by identifying gaps, generating adversarial variations, and updating policies without human intervention. The framework demonstrates a practical path toward resilient, scalable, auditable safety layers that adapt to shifting threats and contexts in real-world dialogue systems.

Abstract

Conversational AI systems require guardrails to prevent harmful outputs, yet existing approaches use static rules that cannot adapt to new threats or deployment contexts. We introduce Lattice, a framework for self-constructing and continuously improving guardrails. Lattice operates in two stages: construction builds initial guardrails from labeled examples through iterative simulation and optimization; continuous improvement autonomously adapts deployed guardrails through risk assessment, adversarial testing, and consolidation. Evaluated on the ProsocialDialog dataset, Lattice achieves 91% F1 on held-out data, outperforming keyword baselines by 43pp, LlamaGuard by 25pp, and NeMo by 4pp. The continuous improvement stage achieves 7pp F1 improvement on cross-domain data through closed-loop optimization. Our framework shows that effective guardrails can be self-constructed through iterative optimization.

Figures (6)

• Figure 1: Two-stage framework architecture.Construction stage (top) generates initial guardrails from labeled conversations through three iterative steps: (1) Conversation Simulation tests current guardrails on synthetic dialogues; (2) Performance Evaluation computes precision, recall, and F1; (3) Guardrail Optimization creates, deletes, tightens, loosens, or clusters guardrails based on false positives and false negatives. Continuous improvement stage (bottom) adapts deployed guardrails to unlabeled conversations through four steps: (1) Risk Assessment identifies coverage gaps via dual-check evaluation; (2) Case Expansion generates adversarial variations; (3) Guardrail Optimization updates policies; (4) Performance Evaluation validates changes and reverts if performance degrades.
• Figure 2: Iterative construction from 100 labeled examples. F1 score (left axis, red) and guardrail count (right axis, blue) across 10 construction iterations. F1 improves from 82% (iteration 0, 23 guardrails) to 93% (iteration 7, 6 guardrails), exceeding the early stopping threshold ($F_1 \geq 0.90$, black dotted line) at iteration 4. Guardrail consolidation reduces the set from 23 to 6 policies through iterative clustering while maintaining performance. The system exhibits convergence with non-monotonic but improving F1 trajectory.
• Figure 3: Sample guardrail: Illegal, unsafe, and unethical behavior monitoring. This guardrail was generated by Lattice during construction to detect conversations involving illegal activities, unsafe advice, or unethical requests. The prompt includes explicit triggering conditions (e.g., planning illegal acts, seeking harmful advice) and exclusion rules (e.g., educational discussions, hypothetical scenarios). The structured format enables consistent evaluation across conversations.
• Figure 4: Sample guardrail: Inappropriate content detection and requests. This guardrail identifies conversations with inappropriate content or requests for harmful material. Generated during construction, it demonstrates how Lattice creates specific triggering patterns (e.g., explicit content requests, inappropriate relationship advice) while avoiding over-flagging through carefully defined exclusions (e.g., medical discussions, educational content).
• Figure : Construction Stage: Iterative Guardrail Optimization