Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GREATERPROMPT: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimization

Wenliang Zheng, Sarkar Snigdha Sarathi Das, Yusen Zhang, Rui Zhang

TL;DR

The paper addresses fragmentation and accessibility barriers in prompt optimization for LLMs. It introduces GreaTerPrompt, a unified open-source framework that consolidates five optimization methods across text-based feedback and gradient-based signals, with support for both local and API-based models and a web UI. Key contributions include a unified API, customizable task exemplars and loss functions, and empirical results showing competitive performance on BBH and GSM8K across model scales. This framework broadens practical deployment of prompt optimization and invites ongoing community contributions.

Abstract

LLMs have gained immense popularity among researchers and the general public for its impressive capabilities on a variety of tasks. Notably, the efficacy of LLMs remains significantly dependent on the quality and structure of the input prompts, making prompt design a critical factor for their performance. Recent advancements in automated prompt optimization have introduced diverse techniques that automatically enhance prompts to better align model outputs with user expectations. However, these methods often suffer from the lack of standardization and compatibility across different techniques, limited flexibility in customization, inconsistent performance across model scales, and they often exclusively rely on expensive proprietary LLM APIs. To fill in this gap, we introduce GREATERPROMPT, a novel framework that democratizes prompt optimization by unifying diverse methods under a unified, customizable API while delivering highly effective prompts for different tasks. Our framework flexibly accommodates various model scales by leveraging both text feedback-based optimization for larger LLMs and internal gradient-based optimization for smaller models to achieve powerful and precise prompt improvements. Moreover, we provide a user-friendly Web UI that ensures accessibility for non-expert users, enabling broader adoption and enhanced performance across various user groups and application scenarios. GREATERPROMPT is available at https://github.com/psunlpgroup/GreaterPrompt via GitHub, PyPI, and web user interfaces.

GREATERPROMPT: A Unified, Customizable, and High-Performing Open-Source Toolkit for Prompt Optimization

TL;DR

The paper addresses fragmentation and accessibility barriers in prompt optimization for LLMs. It introduces GreaTerPrompt, a unified open-source framework that consolidates five optimization methods across text-based feedback and gradient-based signals, with support for both local and API-based models and a web UI. Key contributions include a unified API, customizable task exemplars and loss functions, and empirical results showing competitive performance on BBH and GSM8K across model scales. This framework broadens practical deployment of prompt optimization and invites ongoing community contributions.

Abstract

LLMs have gained immense popularity among researchers and the general public for its impressive capabilities on a variety of tasks. Notably, the efficacy of LLMs remains significantly dependent on the quality and structure of the input prompts, making prompt design a critical factor for their performance. Recent advancements in automated prompt optimization have introduced diverse techniques that automatically enhance prompts to better align model outputs with user expectations. However, these methods often suffer from the lack of standardization and compatibility across different techniques, limited flexibility in customization, inconsistent performance across model scales, and they often exclusively rely on expensive proprietary LLM APIs. To fill in this gap, we introduce GREATERPROMPT, a novel framework that democratizes prompt optimization by unifying diverse methods under a unified, customizable API while delivering highly effective prompts for different tasks. Our framework flexibly accommodates various model scales by leveraging both text feedback-based optimization for larger LLMs and internal gradient-based optimization for smaller models to achieve powerful and precise prompt improvements. Moreover, we provide a user-friendly Web UI that ensures accessibility for non-expert users, enabling broader adoption and enhanced performance across various user groups and application scenarios. GREATERPROMPT is available at https://github.com/psunlpgroup/GreaterPrompt via GitHub, PyPI, and web user interfaces.

Paper Structure

This paper contains 15 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Prompt Optimization Algorithms
  4. Prompt Optimization Services/Libraries
  5. Evaluation Metrics and Datasets for Prompt Optimization
  6. GreaTerPrompt
  7. Unified Implementation
  8. User Customization
  9. High-Performing Prompts
  10. Usage Examples
  11. Python Package
  12. User Friendly Web Interface
  13. Conclusion
  14. GSM8K Results
  15. Optimized Prompts

Figures (1)

  • Figure 2: Screenshot of Web UI for GreaTerPrompt. Optimizer list is on the top left bar, bottom left bar is parameter settings for each optimizer. On the main area, there is a textbox for the model path input, and an area to upload user's prompt data. "P Extractor" is a system prompt for GReaTer optimizer to extract answer to calculate loss.