Can we only use guideline instead of shot in prompt?

TL;DR

The FGT framework is proposed to automatically learn task-specific guidelines from dataset consisting of Feedback, Guideline, and Tree-gather agents to achieve superior performance across multiple tasks, thereby highlighting the effectiveness of using the guidelines in prompt.

Abstract

Currently, prompting techniques can be mainly divided into two categories:1)shot method implicitly inspires the model to answer the question by mimicing the steps in the given example, e.g., the few-shot CoT. 2) Guideline method explicitly instructs the model to reason by following guidelines, which contains succinct and concise task-specific knowledge. Shot method is prone to difficulties in terms of selection of shots type, the number of shots, and the design of the reasoning steps, so a question arises: can we only use guideline instead of shot in the prompt? To this end, we propose the FGT framework to automatically learn task-specific guidelines from dataset consisting of Feedback, Guideline, and Tree-gather agents. First, the feedback agent is designed to evaluate the outcomes, both right and wrong, of each Q&A to gather insights guiding more effective optimization strategies. Next, the guideline agent is tasked with deriving guidelines from each piece of feedback and storing them in local memory. Lastly, the tree-gather agent aggregates all guidelines hierarchically through a tree structure, ultimately obtaining all unduplicated guidelines from a global perspective. In addition, we induce the model to generate intermediate processes to ensure the reasoning consistent with the guidelines. Experimental results demonstrate that our approach achieves superior performance across multiple tasks, thereby highlighting the effectiveness of using the guidelines in prompt.

Figures (8)

• Figure 1: It's commonly using the shot or shot with guideline to assist the LLM to answer the question. Can we only use the guideline in prompt, to achieve comparable performance?
• Figure 2: Overview of our proposed FGT framework for learning guidelines from the data,which consists of three agents:feedback agent,guideline agent and tree-gather agent. It takes the Q&A pair as input, and generates the final prompts with guidelines, with the output of each step stored in memory for the next stage. Task-prompt describes the task in one sentence clearly and concisely.
• Figure 3: Details of our feedback agent, which includes the judgement and the analysis steps.
• Figure 4: Details of our guideline agent,which involves the discussion,design and the guideline parts.
• Figure 5: Illustration of our proposed tree-gather agent.(a) shows the gathering process using the tree structure, where it can be varied to one step of direct combining as k increases. (b) compares the way of gathering and the optimization trajectory from an optimization perspective, reflecting the global view of our approach.