Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification

Peipei Wei, Dimitris Dimitriadis, Yan Xu, Mingwei Shen

TL;DR

We introduce PRINCIPLE-BASED PROMPTING, a multi-agent prompting framework that auto-generates task-specific principles from demonstrations, consolidates them with a finalizer, and uses the resulting principle to perform zero-shot text classification. Across five datasets and multiple LLMs, this approach yields large macro-F1 gains over vanilla prompting and strong ICL baselines, while reducing inference costs relative to multi-step prompts and fine-tuning. Ablation studies show the importance of label information and cooperative multi-agent generation for high-quality principles, and the method approaches or surpasses few-shot ICL in many settings with substantially shorter prompts. The work suggests practical benefits for low-resource or cost-constrained regimes and points to future integrations with retrieval-augmented methods and automated SOP generation.

Abstract

We present PRINCIPLE-BASED PROMPTING, a simple but effective multi-agent prompting strategy for text classification. It first asks multiple LLM agents to independently generate candidate principles based on analysis of demonstration samples with or without labels, consolidates them into final principles via a finalizer agent, and then sends them to a classifier agent to perform downstream classification tasks. Extensive experiments on binary and multi-class classification datasets with different sizes of LLMs show that our approach not only achieves substantial performance gains (1.55% - 19.37%) over zero-shot prompting on macro-F1 score but also outperforms other strong baselines (CoT and stepback prompting). Principles generated by our approach help LLMs perform better on classification tasks than human crafted principles on two private datasets. Our multi-agent PRINCIPLE-BASED PROMPTING approach also shows on-par or better performance compared to demonstration-based few-shot prompting approaches, yet with substantially lower inference costs. Ablation studies show that label information and the multi-agent cooperative LLM framework play an important role in generating high-quality principles to facilitate downstream classification tasks.

Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification

TL;DR

We introduce PRINCIPLE-BASED PROMPTING, a multi-agent prompting framework that auto-generates task-specific principles from demonstrations, consolidates them with a finalizer, and uses the resulting principle to perform zero-shot text classification. Across five datasets and multiple LLMs, this approach yields large macro-F1 gains over vanilla prompting and strong ICL baselines, while reducing inference costs relative to multi-step prompts and fine-tuning. Ablation studies show the importance of label information and cooperative multi-agent generation for high-quality principles, and the method approaches or surpasses few-shot ICL in many settings with substantially shorter prompts. The work suggests practical benefits for low-resource or cost-constrained regimes and points to future integrations with retrieval-augmented methods and automated SOP generation.

Abstract

We present PRINCIPLE-BASED PROMPTING, a simple but effective multi-agent prompting strategy for text classification. It first asks multiple LLM agents to independently generate candidate principles based on analysis of demonstration samples with or without labels, consolidates them into final principles via a finalizer agent, and then sends them to a classifier agent to perform downstream classification tasks. Extensive experiments on binary and multi-class classification datasets with different sizes of LLMs show that our approach not only achieves substantial performance gains (1.55% - 19.37%) over zero-shot prompting on macro-F1 score but also outperforms other strong baselines (CoT and stepback prompting). Principles generated by our approach help LLMs perform better on classification tasks than human crafted principles on two private datasets. Our multi-agent PRINCIPLE-BASED PROMPTING approach also shows on-par or better performance compared to demonstration-based few-shot prompting approaches, yet with substantially lower inference costs. Ablation studies show that label information and the multi-agent cooperative LLM framework play an important role in generating high-quality principles to facilitate downstream classification tasks.

Paper Structure

This paper contains 27 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Demonstration and label relationship
  4. Number of demonstrations
  5. Single-Agent vs. Multi-Agent LLM Framework
  6. Methods
  7. Principle Generation
  8. Principle Consolidation
  9. Text Classification
  10. Baselines
  11. Vanilla Prompting
  12. CoT Prompting
  13. Stepback Prompting
  14. Principle Single-Agent
  15. Finetuning
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Pipeline and Multiagent illustrations of PRINCIPLE-BASED PROMPTING
  • Figure 2: Comparison of input token lengths between principle-based and few-shot vanilla prompting approaches. Stars on each line indicate where the input token length of the principle-based multi-agent consolidation approach corresponds to different n-shot settings (where n ranges from 1 to 8)
  • Figure 3: Effects of label information in sampled demonstrations on generating high-quality principles for downstream classification
  • Figure 4: Effects of single vs multi-agent in generating high-quality principles for downstream classification task