Automatic Prompt Generation via Adaptive Selection of Prompting Techniques

TL;DR

This work addresses the challenge of manual prompt engineering by introducing an adaptive prompt-generation framework driven by a knowledge base that maps semantically defined task clusters to prompting techniques. It constructs task clusters via semantic-space representations and later links each cluster to a curated set of techniques, enabling dynamic, task-adapted prompt creation from abstract user descriptions. Empirical evaluation on 23 BIG-Bench Extra Hard tasks demonstrates that the proposed method outperforms standard prompts and a leading automatic prompt generator in both arithmetic and harmonic mean performance, with further gains when temperature is tuned per task. The approach offers a scalable, domain-agnostic foundation for standardizing prompt design and broadening access to LLM capabilities across diverse user groups and domains.

Abstract

Prompt engineering is crucial for achieving reliable and effective outputs from large language models (LLMs), but its design requires specialized knowledge of prompting techniques and a deep understanding of target tasks. To address this challenge, we propose a novel method that adaptively selects task-appropriate prompting techniques based on users' abstract task descriptions and automatically generates high-quality prompts without relying on pre-existing templates or frameworks. The proposed method constructs a knowledge base that associates task clusters, characterized by semantic similarity across diverse tasks, with their corresponding prompting techniques. When users input task descriptions, the system assigns them to the most relevant task cluster and dynamically generates prompts by integrating techniques drawn from the knowledge base. An experimental evaluation of the proposed method on 23 tasks from BIG-Bench Extra Hard (BBEH) demonstrates superior performance compared with standard prompts and existing automatic prompt-generation tools, as measured by both arithmetic and harmonic mean scores. This research establishes a foundation for streamlining and standardizing prompt creation, enabling non-experts to effectively leverage LLMs.

Figures (29)

• Figure 1: Processing flow diagram of Knowledge Base Construction Phase
• Figure 2: Processing flow of the Prompt Generation Phase
• Figure 3: Box plots of score distributions (Tasks 1-6)
• Figure 4: Box plots of score distributions (Tasks 7-12)
• Figure 5: Box plots of score distributions (Tasks 13-18)