Towards more Contextual Agents: An extractor-Generator Optimization Framework

TL;DR

The paper addresses the problem of domain-specific performance gaps in LLM-based agents caused by insufficient contextual knowledge. It introduces the Extractor-Generator Framework, a two-stage approach comprising Features Extraction to build contextual feature representations and Prompt-Components Generation to craft optimized prompts, augmented by a self-improvement loop. Empirical results across five domains show the framework yields significant gains over standard prompting pipelines, with notable performance in e-commerce (88.1% relevancy) and strong results in finance and healthcare. The approach offers a scalable, automated path to robust, context-aware agent systems and can extend to multi-stage workflows in diverse settings.

Abstract

Large Language Model (LLM)-based agents have demonstrated remarkable success in solving complex tasks across a wide range of general-purpose applications. However, their performance often degrades in context-specific scenarios, such as specialized industries or research domains, where the absence of domain-relevant knowledge leads to imprecise or suboptimal outcomes. To address this challenge, our work introduces a systematic approach to enhance the contextual adaptability of LLM-based agents by optimizing their underlying prompts-critical components that govern agent behavior, roles, and interactions. Manually crafting optimized prompts for context-specific tasks is labor-intensive, error-prone, and lacks scalability. In this work, we introduce an Extractor-Generator framework designed to automate the optimization of contextual LLM-based agents. Our method operates through two key stages: (i) feature extraction from a dataset of gold-standard input-output examples, and (ii) prompt generation via a high-level optimization strategy that iteratively identifies underperforming cases and applies self-improvement techniques. This framework substantially improves prompt adaptability by enabling more precise generalization across diverse inputs, particularly in context-specific tasks where maintaining semantic consistency and minimizing error propagation are critical for reliable performance. Although developed with single-stage workflows in mind, the approach naturally extends to multi-stage workflows, offering broad applicability across various agent-based systems. Empirical evaluations demonstrate that our framework significantly enhances the performance of prompt-optimized agents, providing a structured and efficient approach to contextual LLM-based agents.

Figures (3)

• Figure 1: Architecture of Features Extraction and Prompt-Components Generation.
• Figure 2: Features Extraction on gold data.
• Figure 3: Components Generation using Extracted Features and Self-Improvement.