Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policies

TL;DR

PBSuite introduces a dynamic evaluation suite for assessing LLM adherence to custom, industry-grounded behavioral policies in multi-turn conversations. It combines a diverse dataset of 300 policies across 30 industries with an adaptive multi-turn evaluation framework based on X-Teaming to stress-test compliance under adversarial interactions. Experiments show strong single-turn adherence but substantial drops in multi-turn settings (up to 84% failure), underscoring limitations of current alignment and moderation for pluralistic, domain-specific requirements. By providing both the dataset and the evaluation framework, PBSuite establishes a foundation for future work in robust, context-aware pluralistic alignment in real-world deployments.

Abstract

Large language models (LLMs) are typically aligned to a universal set of safety and usage principles intended for broad public acceptability. Yet, real-world applications of LLMs often take place within organizational ecosystems shaped by distinctive corporate policies, regulatory requirements, use cases, brand guidelines, and ethical commitments. This reality highlights the need for rigorous and comprehensive evaluation of LLMs with pluralistic alignment goals, an alignment paradigm that emphasizes adaptability to diverse user values and needs. In this work, we present PLURALISTIC BEHAVIOR SUITE (PBSUITE), a dynamic evaluation suite designed to systematically assess LLMs' capacity to adhere to pluralistic alignment specifications in multi-turn, interactive conversations. PBSUITE consists of (1) a diverse dataset of 300 realistic LLM behavioral policies, grounded in 30 industries; and (2) a dynamic evaluation framework for stress-testing model compliance with custom behavioral specifications under adversarial conditions. Using PBSUITE, We find that leading open- and closed-source LLMs maintain robust adherence to behavioral policies in single-turn settings (less than 4% failure rates), but their compliance weakens substantially in multi-turn adversarial interactions (up to 84% failure rates). These findings highlight that existing model alignment and safety moderation methods fall short in coherently enforcing pluralistic behavioral policies in real-world LLM interactions. Our work contributes both the dataset and analytical framework to support future research toward robust and context-aware pluralistic alignment techniques.

Figures (9)

• Figure 1: Data creation pipeline. For each industry, we first identify relevant behavioral risk dimensions and assign discrete risk tiers. Based on these tiered dimensions, we construct representative enterprise use cases. Behavior policies are then generated for each use case, conditioned on its associated risk tier configuration.
• Figure 2: Histogram of ratings for behavior policies across different axes.
• Figure 3: Number of successful strategies per behavior and model. Behavior ASR is 1 minus the fraction of uncompromised behaviors by any strategy.
• Figure 4: Intent variation in attack strategies utilized to target gpt-4o.
• Figure 5: Behavior ASR by clusters of strategies for gpt-4o.