Assessing the Performance of Chinese Open Source Large Language Models in Information Extraction Tasks

TL;DR

The paper investigates how Chinese open-source LLMs perform on Information Extraction tasks (NER, RE, EE) under zero-shot and few-shot settings, using ChatGPT as a benchmark. It employs multiple prompting strategies (2-stage NER, Vanilla vs 2-Stage; RE VanillaRE vs QA4RE with/without type constraints; EE via 2-Stage) and standard datasets (MSRA, Weibo, DuIE2.0, DuEE1.0) to assess capabilities. Key findings show ChatGPT consistently outperforms Chinese open-source LLMs across IE tasks; larger open-source models (13B/14B) tend to beat smaller ones in NER, while in RE the Qwen family approaches ChatGPT, yet EE remains challenging for open-source models. The work highlights current gaps in Chinese IE with open-source LLMs and suggests future directions including few-shot fine-tuning and architecture innovations tailored to Chinese IE.

Abstract

Information Extraction (IE) plays a crucial role in Natural Language Processing (NLP) by extracting structured information from unstructured text, thereby facilitating seamless integration with various real-world applications that rely on structured data. Despite its significance, recent experiments focusing on English IE tasks have shed light on the challenges faced by Large Language Models (LLMs) in achieving optimal performance, particularly in sub-tasks like Named Entity Recognition (NER). In this paper, we delve into a comprehensive investigation of the performance of mainstream Chinese open-source LLMs in tackling IE tasks, specifically under zero-shot conditions where the models are not fine-tuned for specific tasks. Additionally, we present the outcomes of several few-shot experiments to further gauge the capability of these models. Moreover, our study includes a comparative analysis between these open-source LLMs and ChatGPT, a widely recognized language model, on IE performance. Through meticulous experimentation and analysis, we aim to provide insights into the strengths, limitations, and potential enhancements of existing Chinese open-source LLMs in the domain of Information Extraction within the context of NLP.