EL4NER: Ensemble Learning for Named Entity Recognition via Multiple Small-Parameter Large Language Models

TL;DR

EL4NER tackles NER under the in-context learning paradigm by ensembling multiple open-source, small-parameter LLMs in a four-stage pipeline. It introduces a span-level demonstration retrieval driven by SpanSim$$(\hat{S}, \hat{S}^c) = \frac{\sum_{s_x \in \hat{S}} w(s_x) \max_{s_c \in \hat{S}^c} \mathrm{Sim}(s_x,s_c)}{\sum_{s_x \in \hat{S}} w(s_x)}$$ and POS-based span weighting to improve demonstration quality, followed by span extraction, span classification via voting, and a type-verification-based filtering stage. The method, using about 37B total parameter count across three open-source backbones, achieves competitive or state-of-the-art performance among ICL-based NER methods on several datasets (ACE05, GENIA, WNUT17) while reducing reliance on closed-source, large LLMs. Ablation studies show the critical value of each component and discuss practical considerations like verifier choice, backbone count, and demonstrated examples, highlighting both the gains and limitations of the approach.

Abstract

In-Context Learning (ICL) technique based on Large Language Models (LLMs) has gained prominence in Named Entity Recognition (NER) tasks for its lower computing resource consumption, less manual labeling overhead, and stronger generalizability. Nevertheless, most ICL-based NER methods depend on large-parameter LLMs: the open-source models demand substantial computational resources for deployment and inference, while the closed-source ones incur high API costs, raise data-privacy concerns, and hinder community collaboration. To address this question, we propose an Ensemble Learning Method for Named Entity Recognition (EL4NER), which aims at aggregating the ICL outputs of multiple open-source, small-parameter LLMs to enhance overall performance in NER tasks at less deployment and inference cost. Specifically, our method comprises three key components. First, we design a task decomposition-based pipeline that facilitates deep, multi-stage ensemble learning. Second, we introduce a novel span-level sentence similarity algorithm to establish an ICL demonstration retrieval mechanism better suited for NER tasks. Third, we incorporate a self-validation mechanism to mitigate the noise introduced during the ensemble process. We evaluated EL4NER on multiple widely adopted NER datasets from diverse domains. Our experimental results indicate that EL4NER surpasses most closed-source, large-parameter LLM-based methods at a lower parameter cost and even attains state-of-the-art (SOTA) performance among ICL-based methods on certain datasets. These results show the parameter efficiency of EL4NER and underscore the feasibility of employing open-source, small-parameter LLMs within the ICL paradigm for NER tasks.

Figures (4)

• Figure 1: Illustration for the comparison of EL4NER with previous ICL-based methods. Most of the previous methods use closed-source large-parameter LLMs to perform single-stage ICL to accomplish the NER task, whereas EL4NER employs multiple open-source small-parameter LLMs to perform multi-stage ICL through ensemble learning to integrate the inference results of these LLMs by means of union and vote.
• Figure 2: Overview of the proposed EL4NER, which adopts a multi-stage ensemble learning method for NER. It includes four stages: (1) Demonstration Retrieval, selecting span-relevant samples for prompt construction; (2) Span Extraction, identifying potential entity spans; (3) Span Classification, assigning entity types to the extracted spans; (4) Type Verification, filtering out incorrectly classified spans. Each stage leverages multiple small-parameter LLMs to enhance robustness and effect of ICL.
• Figure 3: Overall comparison of variants.
• Figure 4: EL4NER performance vs. #demonstrations in the three datasets.