Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReFineG: Synergizing Small Supervised Models and LLMs for Low-Resource Grounded Multimodal NER

Jielong Tang, Shuang Wang, Zhenxing Wang, Jianxing Yu, Jian Yin

TL;DR

ReFineG tackles the challenge of low-resource grounded multimodal NER by pairing small supervised models with frozen multimodal LLMs in a three-stage workflow. It uses domain knowledge-aware data synthesis to transfer knowledge to a compact model, followed by an uncertainty-based refinement where uncertain cases are handed to an LLM with chain-of-thought prompts. For grounding, it employs multimodal in-context learning with top-K cross-modal example selection to improve visual region grounding. Across Twitter-GMNER and CCKS-GMNER, ReFineG achieves strong results, ranking second on the CCKS2025 GMNER Shared Task, demonstrating effective annotation efficiency and cross-domain applicability.

Abstract

Grounded Multimodal Named Entity Recognition (GMNER) extends traditional NER by jointly detecting textual mentions and grounding them to visual regions. While existing supervised methods achieve strong performance, they rely on costly multimodal annotations and often underperform in low-resource domains. Multimodal Large Language Models (MLLMs) show strong generalization but suffer from Domain Knowledge Conflict, producing redundant or incorrect mentions for domain-specific entities. To address these challenges, we propose ReFineG, a three-stage collaborative framework that integrates small supervised models with frozen MLLMs for low-resource GMNER. In the Training Stage, a domain-aware NER data synthesis strategy transfers LLM knowledge to small models with supervised training while avoiding domain knowledge conflicts. In the Refinement Stage, an uncertainty-based mechanism retains confident predictions from supervised models and delegates uncertain ones to the MLLM. In the Grounding Stage, a multimodal context selection algorithm enhances visual grounding through analogical reasoning. In the CCKS2025 GMNER Shared Task, ReFineG ranked second with an F1 score of 0.6461 on the online leaderboard, demonstrating its effectiveness with limited annotations.

ReFineG: Synergizing Small Supervised Models and LLMs for Low-Resource Grounded Multimodal NER

TL;DR

ReFineG tackles the challenge of low-resource grounded multimodal NER by pairing small supervised models with frozen multimodal LLMs in a three-stage workflow. It uses domain knowledge-aware data synthesis to transfer knowledge to a compact model, followed by an uncertainty-based refinement where uncertain cases are handed to an LLM with chain-of-thought prompts. For grounding, it employs multimodal in-context learning with top-K cross-modal example selection to improve visual region grounding. Across Twitter-GMNER and CCKS-GMNER, ReFineG achieves strong results, ranking second on the CCKS2025 GMNER Shared Task, demonstrating effective annotation efficiency and cross-domain applicability.

Abstract

Grounded Multimodal Named Entity Recognition (GMNER) extends traditional NER by jointly detecting textual mentions and grounding them to visual regions. While existing supervised methods achieve strong performance, they rely on costly multimodal annotations and often underperform in low-resource domains. Multimodal Large Language Models (MLLMs) show strong generalization but suffer from Domain Knowledge Conflict, producing redundant or incorrect mentions for domain-specific entities. To address these challenges, we propose ReFineG, a three-stage collaborative framework that integrates small supervised models with frozen MLLMs for low-resource GMNER. In the Training Stage, a domain-aware NER data synthesis strategy transfers LLM knowledge to small models with supervised training while avoiding domain knowledge conflicts. In the Refinement Stage, an uncertainty-based mechanism retains confident predictions from supervised models and delegates uncertain ones to the MLLM. In the Grounding Stage, a multimodal context selection algorithm enhances visual grounding through analogical reasoning. In the CCKS2025 GMNER Shared Task, ReFineG ranked second with an F1 score of 0.6461 on the online leaderboard, demonstrating its effectiveness with limited annotations.

Paper Structure

This paper contains 10 sections, 9 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ReFineG Framework
  4. Stage 1: Training with Synthetic Data
  5. Stage 2: Uncertainty-based Refinement
  6. Stage 3: Grounding with Multimodal In-context Learning
  7. Experiments
  8. Experiment Setting
  9. Main Results
  10. Ablation Study.

Figures (4)

  • Figure 1: The comparison of existing approaches and our ReFineG.
  • Figure 2: The overall framework of our ReFineG.
  • Figure 3: The prompt details designed for data synthesis, refinement, and entity grounding.
  • Figure 4: Ablation study results. (a) Training with Synthetic Data (TSD) (b) Uncertainty-based Refinement (UR) (c) Multimodal Examples Selection (MES).