Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification

Zhen Qin, Junru Wu, Jiaming Shen, Tianqi Liu, Xuanhui Wang

TL;DR

LAMPO reframes few-shot ordinal classification as a pairwise preference task where an LLM compares a test instance to each demonstration. The framework aggregates these binary comparisons into an ordinal decision via offline-thresholding, combining an expected-threshold prior with self-supervised thresholds derived from a probing set. It eliminates reliance on internal logits and supports arbitrarily many demonstrations, addressing context-length and ordering-bias issues that hinder traditional in-context learning. Across seven public datasets and two LLMs, LAMPO delivers competitive or superior performance, with notable gains on hard tasks like hate detection, and demonstrates robustness to shot-size variations and black-box LLM usage.

Abstract

We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration examples with the test instance and prompt LLMs to produce the pointwise prediction, our framework uses the LLM as a preference machine that makes a relative comparative decision between the test instance and each demonstration. A self-supervised method is then introduced to aggregate these binary comparisons into the final ordinal decision. LAMPO addresses several limitations inherent in previous methods, including context length constraints, ordering biases, and challenges associated with absolute point-wise estimation. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications (e.g., movie review analysis and hate speech detection). Notably, in certain applications, the improvement can be substantial, exceeding 20% in an absolute term. Moreover, we believe LAMPO represents an interesting addition to the non-parametric application layered on top of LLMs, as it supports black-box LLMs without necessitating the outputting of LLM's internal states (e.g., embeddings), as seen in previous approaches.

LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification

TL;DR

LAMPO reframes few-shot ordinal classification as a pairwise preference task where an LLM compares a test instance to each demonstration. The framework aggregates these binary comparisons into an ordinal decision via offline-thresholding, combining an expected-threshold prior with self-supervised thresholds derived from a probing set. It eliminates reliance on internal logits and supports arbitrarily many demonstrations, addressing context-length and ordering-bias issues that hinder traditional in-context learning. Across seven public datasets and two LLMs, LAMPO delivers competitive or superior performance, with notable gains on hard tasks like hate detection, and demonstrates robustness to shot-size variations and black-box LLM usage.

Abstract

We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration examples with the test instance and prompt LLMs to produce the pointwise prediction, our framework uses the LLM as a preference machine that makes a relative comparative decision between the test instance and each demonstration. A self-supervised method is then introduced to aggregate these binary comparisons into the final ordinal decision. LAMPO addresses several limitations inherent in previous methods, including context length constraints, ordering biases, and challenges associated with absolute point-wise estimation. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications (e.g., movie review analysis and hate speech detection). Notably, in certain applications, the improvement can be substantial, exceeding 20% in an absolute term. Moreover, we believe LAMPO represents an interesting addition to the non-parametric application layered on top of LLMs, as it supports black-box LLMs without necessitating the outputting of LLM's internal states (e.g., embeddings), as seen in previous approaches.
Paper Structure (38 sections, 8 equations, 2 figures, 3 tables)

This paper contains 38 sections, 8 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. In-Context Learning (ICL) for Few-shot Ordinal Classification
  4. Contextual Calibration (CC)
  5. Global Entropy (GlobalE)
  6. Scope
  7. LAMPO
  8. Scoring
  9. Decision Making
  10. A Running Example
  11. Experiments
  12. Datasets
  13. Baselines and Model Configurations
  14. Experimental Results
  15. Results on Twitter.
  16. ...and 23 more sections

Figures (2)

  • Figure 1: An illustration of our LAMPO on a m-way sentiment classification task. The scoring step compares test instance with every demonstration using LLM. The decision making step converts the score to an ordinal label by using thresholds learned offline.
  • Figure 2: An illustration of probing set construction from lu2022fantastically. "PLM" is the same as "LLM" in the context of this paper.