AWED-FiNER: Agents, Web applications, and Expert Detectors for Fine-grained Named Entity Recognition across 36 Languages for 6.6 Billion Speakers

TL;DR

AWED-FiNER introduces an open-source ecosystem for Fine-grained Named Entity Recognition across 36 languages spoken by billions, combining an agentic toolkit, web-based interfaces, and a large collection of 49 expert detectors. The approach leverages a Hugging Face Router API to route multilingual text to specialized models and supports offline deployment on resource-constrained devices. Through fine-tuning on established benchmarks like MultiCoNER2 and FewNERD with state-of-the-art multilingual backbones, AWED-FiNER demonstrates strong Macro-F1 performance and provides a practical deployment platform via Gradio apps. The work emphasizes linguistic equity by including vulnerable languages such as Bodo, Manipuri, Bishnupriya, and Mizo, and positions AWED-FiNER as the first comprehensive pipeline that spans agent tools, interactive web apps, and expert models across a broad language spectrum.

Abstract

We introduce AWED-FiNER, an open-source ecosystem designed to bridge the gap in Fine-grained Named Entity Recognition (FgNER) for 36 global languages spoken by more than 6.6 billion people. While Large Language Models (LLMs) dominate general Natural Language Processing (NLP) tasks, they often struggle with low-resource languages and fine-grained NLP tasks. AWED-FiNER provides a collection of agentic toolkits, web applications, and several state-of-the-art expert models that provides FgNER solutions across 36 languages. The agentic tools enable to route multilingual text to specialized expert models and fetch FgNER annotations within seconds. The web-based platforms provide ready-to-use FgNER annotation service for non-technical users. Moreover, the collection of language specific extremely small sized open-source state-of-the-art expert models facilitate offline deployment in resource contraint scenerios including edge devices. AWED-FiNER covers languages spoken by over 6.6 billion people, including a specific focus on vulnerable languages such as Bodo, Manipuri, Bishnupriya, and Mizo. The resources can be accessed here: Agentic Tool (https://github.com/PrachuryyaKaushik/AWED-FiNER), Web Application (https://hf.co/spaces/prachuryyaIITG/AWED-FiNER), and 49 Expert Detector Models (https://hf.co/collections/prachuryyaIITG/awed-finer).