Auto-Prompting with Retrieval Guidance for Frame Detection in Logistics

TL;DR

The paper tackles frame detection in logistics texts under extreme data scarcity by proposing Auto-Prompting with Retrieval Guidance, a pipeline that integrates retrieval-augmented generation, few-shot prompting, chain-of-thought reasoning, and Auto-CoT within an autonomous prompt-optimizer loop. An LLM-based prompt optimizer iteratively refines prompts using retrieved exemplars and self-evaluation, achieving up to 15% accuracy gains over zero-shot or static prompts and generalizing across GPT-4o, Qwen 2.5, and LLaMA 3.1. The key contributions are the integrated RAG-driven prompt construction, Auto-CoT-enabled reasoning exemplars, and a model-agnostic framework that reduces prompting effort while maintaining robustness in a real-world, low-resource domain. This work demonstrates that structured prompt optimization can rival traditional fine-tuning for domain-specific NLP tasks, enabling scalable deployment in logistics and similar industrial settings.

Abstract

Prompt engineering plays a critical role in adapting large language models (LLMs) to complex reasoning and labeling tasks without the need for extensive fine-tuning. In this paper, we propose a novel prompt optimization pipeline for frame detection in logistics texts, combining retrieval-augmented generation (RAG), few-shot prompting, chain-of-thought (CoT) reasoning, and automatic CoT synthesis (Auto-CoT) to generate highly effective task-specific prompts. Central to our approach is an LLM-based prompt optimizer agent that iteratively refines the prompts using retrieved examples, performance feedback, and internal self-evaluation. Our framework is evaluated on a real-world logistics text annotation task, where reasoning accuracy and labeling efficiency are critical. Experimental results show that the optimized prompts - particularly those enhanced via Auto-CoT and RAG - improve real-world inference accuracy by up to 15% compared to baseline zero-shot or static prompts. The system demonstrates consistent improvements across multiple LLMs, including GPT-4o, Qwen 2.5 (72B), and LLaMA 3.1 (70B), validating its generalizability and practical value. These findings suggest that structured prompt optimization is a viable alternative to full fine-tuning, offering scalable solutions for deploying LLMs in domain-specific NLP applications such as logistics.

Figures (3)

• Figure 1: Word cloud of the most frequent tokens in the dataset
• Figure 2: Distribution of sentence lengths (in number of words)
• Figure 3: Overview of the proposed prompt optimization pipeline.