APE: Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM-based Entity Matching

TL;DR

The paper addresses the challenge of identifying informative few-shot demonstrations for LLM prompting in entity matching by proposing APE, an active learning-based interactive tool with a graph UI. APE iteratively samples ambiguous examples, incorporates human annotations, updates prompts, and evaluates performance, using task-agnostic strategies such as random-based and self-consistency-based sampling. The self-consistency approach uses multiple prompt variants and entropy-based uncertainty to select high-value examples for annotation. This tooling-centric contribution aims to reduce manual prompt engineering effort while boosting in-context learning effectiveness, with demonstrations contextualized on an entity-matching dataset and a video demonstration. The work emphasizes practical applicability and outlines future directions for theoretical grounding and experimental validation.

Abstract

Prompt engineering is an iterative procedure often requiring extensive manual effort to formulate suitable instructions for effectively directing large language models (LLMs) in specific tasks. Incorporating few-shot examples is a vital and effective approach to providing LLMs with precise instructions, leading to improved LLM performance. Nonetheless, identifying the most informative demonstrations for LLMs is labor-intensive, frequently entailing sifting through an extensive search space. In this demonstration, we showcase a human-in-the-loop tool called APE (Active Prompt Engineering) designed for refining prompts through active learning. Drawing inspiration from active learning, APE iteratively selects the most ambiguous examples for human feedback, which will be transformed into few-shot examples within the prompt. The demo recording can be found with the submission or be viewed at https://youtu.be/OwQ6MQx53-Y.

Figures (1)

• Figure 1: System Overview