Analyzing and Internalizing Complex Policy Documents for LLM Agents

TL;DR

This work tackles the problem of heavy prompt overhead from long policy documents in LLM-based agents. It introduces CC-Gen, a controllable-complexity benchmark, and CAP-CPT, a Category-Aware Policy Continued Pretraining framework that organizes policy specifications into four categories and generates targeted data for CPT. The authors show that workflow-level policy complexity is a major challenge and demonstrate that CAP-CPT reduces data requirements while improving robustness across diverse task settings, achieving substantial input compression on CC-Gen and benefiting tau-bench evaluations. The approach yields consistent performance gains, narrows complexity-related gaps, and provides a scalable path toward reliable, policy-guided LLM agents in real-world applications.

Abstract

Large Language Model (LLM)-based agentic systems rely on in-context policy documents encoding diverse business rules. As requirements grow, these documents expand rapidly, causing high computational overhead. This motivates developing internalization methods that embed policy documents into model priors while preserving performance. Prior prompt compression work targets generic prompts, but agentic policy documents span multiple complexity levels and require deeper reasoning, making internalization harder. We introduce CC-Gen, an agentic benchmark generator with Controllable Complexity across four levels, enabling systematic evaluation of agents' ability to handle complexity and offering a unified framework for assessing policy internalization. Our analysis shows that complex policy specifications governing workflows pose major reasoning challenges. Supporting internalization with gold user agent interaction trajectories containing chain-of-thought (CoT) annotations via supervised fine-tuning (SFT) is data-intensive and degrades sharply as policy complexity increases. To mitigate data and reasoning burdens, we propose Category-Aware Policy Continued Pretraining (CAP-CPT). Our automated pipeline parses policy documents to extract key specifications, grouping them into factual, behavioral, and conditional categories, and isolating complex conditions that drive workflow complexity. This guides targeted data synthesis and enables agents to internalize policy information through an autoregressive pretraining loss. Experiments show CAP-CPT improves SFT baselines in all settings, with up to 41% and 22% gains on Qwen-3-32B, achieving 97.3% prompt length reduction on CC-Gen and further enhancing tau-Bench with minimal SFT data.

Figures (9)

• Figure 1: Even state-of-the-art LLM-based agents fail to reliably follow policy documents, and our analysis shows that certain policy specifications are inherently complex, imposing substantial reasoning demands. These observations motivate the central research questions we investigate in this paper. A more detailed illustration of this failure case is provided in Appendix \ref{['app: Error Examples']}.
• Figure 2: Pipeline for our Category-Aware Policy Continued Pretraining (CAP-CPT).Top: An LLM-centric pipeline analyzes policy documents and categorizes policy specifications into four major types. Bottom: Based on this categorization, we generate targeted training data for each specification type. In particular, scenario-simulation examples address conditional rules that require complex reasoning, helping the model internalize and apply the most challenging policy knowledge.
• Figure 3: Performance curves for internalizing policy documents with varying workflow complexities on Qwen-2.5-32B, comparing the baseline with our method. Our approach consistently outperforms the baseline across all settings and substantially narrows the performance gap in high-complexity and data-sparse scenarios.
• Figure 4: Pipeline of our CC-Gen benchmark generator.
• Figure 5: Performance curves for internalizing policy documents with varying workflow complexities on Qwen-2.5-32B, comparing the baseline with our method. Our approach consistently outperforms the baseline across all settings and substantially narrows the performance gap in high-complexity and data-sparse scenarios. Note that while Qwen-3-32B is a model with stronger prior knowledge, the internalization only yields comparable performance than prompting baseline. See Appendix \ref{['app: explanation']} for explanations.