PerfGuard: A Performance-Aware Agent for Visual Content Generation

TL;DR

PerfGuard tackles planning uncertainty in visual content generation by explicitly modeling tool performance boundaries and integrating execution feedback into planning. It introduces Performance-Aware Selection Modeling (PASM), Adaptive Preference Updating (APU), and Capability-Aligned Planning Optimization (CAPO) to align tool choice, plan generation, and task execution with real-world tool performance. Across generation and editing benchmarks, PerfGuard achieves higher tool-selection accuracy, more reliable outputs, and closer alignment with user intent, validated by qualitative analyses, quantitative metrics, and ablations. The work demonstrates robust improvements and provides open-source code to support reproducibility and further development in multi-agent visual reasoning tasks.

Abstract

The advancement of Large Language Model (LLM)-powered agents has enabled automated task processing through reasoning and tool invocation capabilities. However, existing frameworks often operate under the idealized assumption that tool executions are invariably successful, relying solely on textual descriptions that fail to distinguish precise performance boundaries and cannot adapt to iterative tool updates. This gap introduces uncertainty in planning and execution, particularly in domains like visual content generation (AIGC), where nuanced tool performance significantly impacts outcomes. To address this, we propose PerfGuard, a performance-aware agent framework for visual content generation that systematically models tool performance boundaries and integrates them into task planning and scheduling. Our framework introduces three core mechanisms: (1) Performance-Aware Selection Modeling (PASM), which replaces generic tool descriptions with a multi-dimensional scoring system based on fine-grained performance evaluations; (2) Adaptive Preference Update (APU), which dynamically optimizes tool selection by comparing theoretical rankings with actual execution rankings; and (3) Capability-Aligned Planning Optimization (CAPO), which guides the planner to generate subtasks aligned with performance-aware strategies. Experimental comparisons against state-of-the-art methods demonstrate PerfGuard's advantages in tool selection accuracy, execution reliability, and alignment with user intent, validating its robustness and practical utility for complex AIGC tasks. The project code is available at https://github.com/FelixChan9527/PerfGuard.

Figures (13)

• Figure 1: PerfGuard decomposes user requests into subtasks for iterative visual content generation. By modeling tool performance boundaries via PASM, it selects the most suitable tool in each round to ensure precise alignment between planning, execution, and user intent.
• Figure 2: PerfGuard models tool performance boundaries via PASM to match the most suitable tool for each subtask, maximizing decision efficiency. It further integrates Adaptive Preference Updating to enhance real-world adaptability, and applies CAPO to align planning with performance-aware strategies.
• Figure 3: Comparison of PerfGuard's visual results across tasks. Top: visualization for complex text-to-image generation. Bottom: visualization for multi-round image editing.
• Figure 4: Comparison of capability matching methods. Our method substantially reduces tool selection errors.
• Figure 5: Ablation on $\eta$ in Eq. \ref{['eq:apu']}. When $\eta = 0.13$, the error rate reaches its minimum of $14.2\%$ at step 800.