Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis

Wenxuan Mu, Jinzhong Ning, Di Zhao, Yijia Zhang

TL;DR

KDR-Agent addresses multi-domain, low-resource NER by integrating external knowledge retrieval from Wikipedia, domain-specific disambiguation, and reflective analysis within a two-stage, multi-agent LLM framework. It replaces heavy Retrieval-based demonstrations with concise natural-language type definitions and static contrastive demonstrations, while adding planning, knowledge retrieval, and disambiguation agents to compensate for domain gaps. The Reflective Analysis stage then performs structured self-evaluation to guide a final corrective pass, yielding robust improvements across ten datasets and multiple backbones. Empirical results show consistent gains over zero-shot and few-shot baselines, with the strongest improvements in specialized domains such as Biomedical and Social Media, highlighting the framework’s generalizability and practical impact in low-resource settings.

Abstract

In-context learning (ICL) with large language models (LLMs) has emerged as a promising paradigm for named entity recognition (NER) in low-resource scenarios. However, existing ICL-based NER methods suffer from three key limitations: (1) reliance on dynamic retrieval of annotated examples, which is problematic when annotated data is scarce; (2) limited generalization to unseen domains due to the LLM's insufficient internal domain knowledge; and (3) failure to incorporate external knowledge or resolve entity ambiguities. To address these challenges, we propose KDR-Agent, a novel multi-agent framework for multi-domain low-resource in-context NER that integrates Knowledge retrieval, Disambiguation, and Reflective analysis. KDR-Agent leverages natural-language type definitions and a static set of entity-level contrastive demonstrations to reduce dependency on large annotated corpora. A central planner coordinates specialized agents to (i) retrieve factual knowledge from Wikipedia for domain-specific mentions, (ii) resolve ambiguous entities via contextualized reasoning, and (iii) reflect on and correct model predictions through structured self-assessment. Experiments across ten datasets from five domains demonstrate that KDR-Agent significantly outperforms existing zero-shot and few-shot ICL baselines across multiple LLM backbones. The code and data can be found at https://github.com/MWXGOD/KDR-Agent.

A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis

TL;DR

KDR-Agent addresses multi-domain, low-resource NER by integrating external knowledge retrieval from Wikipedia, domain-specific disambiguation, and reflective analysis within a two-stage, multi-agent LLM framework. It replaces heavy Retrieval-based demonstrations with concise natural-language type definitions and static contrastive demonstrations, while adding planning, knowledge retrieval, and disambiguation agents to compensate for domain gaps. The Reflective Analysis stage then performs structured self-evaluation to guide a final corrective pass, yielding robust improvements across ten datasets and multiple backbones. Empirical results show consistent gains over zero-shot and few-shot baselines, with the strongest improvements in specialized domains such as Biomedical and Social Media, highlighting the framework’s generalizability and practical impact in low-resource settings.

Abstract

In-context learning (ICL) with large language models (LLMs) has emerged as a promising paradigm for named entity recognition (NER) in low-resource scenarios. However, existing ICL-based NER methods suffer from three key limitations: (1) reliance on dynamic retrieval of annotated examples, which is problematic when annotated data is scarce; (2) limited generalization to unseen domains due to the LLM's insufficient internal domain knowledge; and (3) failure to incorporate external knowledge or resolve entity ambiguities. To address these challenges, we propose KDR-Agent, a novel multi-agent framework for multi-domain low-resource in-context NER that integrates Knowledge retrieval, Disambiguation, and Reflective analysis. KDR-Agent leverages natural-language type definitions and a static set of entity-level contrastive demonstrations to reduce dependency on large annotated corpora. A central planner coordinates specialized agents to (i) retrieve factual knowledge from Wikipedia for domain-specific mentions, (ii) resolve ambiguous entities via contextualized reasoning, and (iii) reflect on and correct model predictions through structured self-assessment. Experiments across ten datasets from five domains demonstrate that KDR-Agent significantly outperforms existing zero-shot and few-shot ICL baselines across multiple LLM backbones. The code and data can be found at https://github.com/MWXGOD/KDR-Agent.

Paper Structure

This paper contains 28 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. In-Context NER
  4. LLM-Based Multi-Agent IE Systems
  5. Methodology
  6. Task Formulation
  7. Overall Framework of KDR-Agent
  8. Stage 1: Knowledge In-context Construction
  9. Natural-Language Type Definitions
  10. Static Few-Shot Contrastive Demonstrations
  11. Central LLM Planner
  12. Knowledge Retrieval Agent
  13. Disambiguation Agent
  14. Stage 2: Reflection & Correction
  15. Initial Inference & Multi-Knowledge Integration
  16. ...and 13 more sections

Figures (2)

  • Figure 1: The overall architecture of the KDR-Agent framework, consisting of two stages: (a) Knowledge In-context Construction and (b) Reflection & Correction. The top illustrates the prompt construction process guided by external agents for knowledge retrieval and disambiguation, incorporating entity type definitions and contrastive demonstrations. The bottom shows the inference and reflective correction workflow based on predefined error criteria.
  • Figure 2: F1 performance of KDR-Agent on three datasets with different sizes of Qwen LLM backbones.