Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MODP: Multi Objective Directional Prompting

Aashutosh Nema, Samaksh Gulati, Evangelos Giakoumakis, Bipana Thapaliya

TL;DR

MODP reframes prompt engineering as a multi-objective directional optimization that jointly considers task-specific objectives and an LLM's intrinsic behavior. By defining objectives, constructing representative samples, weighting criteria, and iterating prompts, the framework yields robust, production-ready prompts and improves performance on both synthetic and enterprise data, demonstrated by a 26% gain and production deployment in Dell’s Next Best Action tool. The approach emphasizes a structured, metric-driven process with human-in-the-loop considerations to manage hallucinations, toxicity, and output adherence, addressing core limitations of prior prompt engineering work. In practice, MODP offers a scalable path to reliable LLM deployment across tasks and models, with potential extensions to multi-agent systems, long-context reasoning, and Pareto-front optimization for stopping criteria.

Abstract

Recent advances in large language models (LLMs) have led to their popularity across multiple use-cases. However, prompt engineering, the process for optimally utilizing such models, remains approximation-driven and subjective. Most of the current research on prompt engineering focuses on task-specific optimization, while neglecting the behavior of the LLM under consideration during prompt development. This paper introduces MODP -- Multi Objective Directional Prompting, a framework based on two key concepts: 1) multi-objectivity: the importance of considering an LLM's intrinsic behavior as an additional objective in prompt development, and 2) directional prompting: a metrics-driven method for prompt engineering to ensure development of robust and high-precision prompts. We demonstrate the effectiveness of our proposed ideas on a summarization task, using a synthetically created dataset, achieving a 26% performance gain over initial prompts. Finally, we apply MODP to develop prompts for Dell's Next Best Action support tool, which is now in production and is used by more than 10,000 internal support agents and serving millions of customers worldwide.

MODP: Multi Objective Directional Prompting

TL;DR

MODP reframes prompt engineering as a multi-objective directional optimization that jointly considers task-specific objectives and an LLM's intrinsic behavior. By defining objectives, constructing representative samples, weighting criteria, and iterating prompts, the framework yields robust, production-ready prompts and improves performance on both synthetic and enterprise data, demonstrated by a 26% gain and production deployment in Dell’s Next Best Action tool. The approach emphasizes a structured, metric-driven process with human-in-the-loop considerations to manage hallucinations, toxicity, and output adherence, addressing core limitations of prior prompt engineering work. In practice, MODP offers a scalable path to reliable LLM deployment across tasks and models, with potential extensions to multi-agent systems, long-context reasoning, and Pareto-front optimization for stopping criteria.

Abstract

Recent advances in large language models (LLMs) have led to their popularity across multiple use-cases. However, prompt engineering, the process for optimally utilizing such models, remains approximation-driven and subjective. Most of the current research on prompt engineering focuses on task-specific optimization, while neglecting the behavior of the LLM under consideration during prompt development. This paper introduces MODP -- Multi Objective Directional Prompting, a framework based on two key concepts: 1) multi-objectivity: the importance of considering an LLM's intrinsic behavior as an additional objective in prompt development, and 2) directional prompting: a metrics-driven method for prompt engineering to ensure development of robust and high-precision prompts. We demonstrate the effectiveness of our proposed ideas on a summarization task, using a synthetically created dataset, achieving a 26% performance gain over initial prompts. Finally, we apply MODP to develop prompts for Dell's Next Best Action support tool, which is now in production and is used by more than 10,000 internal support agents and serving millions of customers worldwide.

Paper Structure

This paper contains 36 sections, 1 equation, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges in Prompt Development
  4. LLM Hallucinations
  5. Instruction loss:
  6. The Guesswork of Prompt Design:
  7. Balancing Hallucination and Accuracy
  8. Addressing the guesswork and importance of the human in loop:
  9. Task-Specific Optimization:
  10. Multi-Objective Directional Prompting
  11. Defining Objectives
  12. Finding a Representative Sample
  13. Weighting of Objectives
  14. Iterating Through Prompts
  15. Dataset
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Prompt Objectives
  • Figure 2: Methodology
  • Figure 3: How was the Dataset Created?
  • Figure 4: Category accuracy of different prompts
  • Figure 5: Prompts vs Accuracy