FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarios

TL;DR

Domain-specific NER in data-scarce settings poses a challenge for full fine-tuning. FsPONER investigates few-shot prompt optimization by stratifying a 300-sample dataset and applying three selection strategies (random, TF-IDF, and a hybrid) to construct domain-aware prompts across multiple LLMs. TF-IDF-based FsPONER often yields the strongest domain performance, with GPT-4-32K achieving top results as prompts scale and an approximate 10% F1 gain over fine-tuned baselines in data-scarce industrial scenarios. The work demonstrates that carefully designed, data-efficient prompting can rival or surpass fine-tuning in real-world domain NER, guiding practical deployment under resource and latency constraints.

Abstract

Large Language Models (LLMs) have provided a new pathway for Named Entity Recognition (NER) tasks. Compared with fine-tuning, LLM-powered prompting methods avoid the need for training, conserve substantial computational resources, and rely on minimal annotated data. Previous studies have achieved comparable performance to fully supervised BERT-based fine-tuning approaches on general NER benchmarks. However, none of the previous approaches has investigated the efficiency of LLM-based few-shot learning in domain-specific scenarios. To address this gap, we introduce FsPONER, a novel approach for optimizing few-shot prompts, and evaluate its performance on domain-specific NER datasets, with a focus on industrial manufacturing and maintenance, while using multiple LLMs -- GPT-4-32K, GPT-3.5-Turbo, LLaMA 2-chat, and Vicuna. FsPONER consists of three few-shot selection methods based on random sampling, TF-IDF vectors, and a combination of both. We compare these methods with a general-purpose GPT-NER method as the number of few-shot examples increases and evaluate their optimal NER performance against fine-tuned BERT and LLaMA 2-chat. In the considered real-world scenarios with data scarcity, FsPONER with TF-IDF surpasses fine-tuned models by approximately 10% in F1 score.

Figures (11)

• Figure 1: Overview of FsPONER.
• Figure 2: The few-shot selection process based on TF-IDF vectors.
• Figure 3: The prompt structure for domain-specific NER tasks.
• Figure 4: A pre-processed prompt-completion pair for fine-tuning.
• Figure 5: The principle of LoRA.