Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis

TL;DR

This paper tackles the paucity and lack of diversity in NLU instruction data by introducing Hum, a large-scale synthetic instruction corpus covering IE, open IE, MRC, TC, and an instruction generalist. Hum employs a human-LLM collaborative synthesis framework with three diversification mechanisms—guidelines synthesis, preference rules synthesis, and format variants synthesis—to expand task coverage and prevent overfitting. Through zero-shot evaluation on five NLU datasets and 28 general capability benchmarks across six LLMs, Hum achieves an average 3.1% improvement in NLU performance with minimal or no degradation in other capabilities, and in several cases surpasses train-free baselines including GPT-4 on NLU tasks. The work demonstrates the practicality of large-scale instruction synthesis for improving language understanding while preserving broad competencies, and outlines a path for further expanding data modalities to maintain generalization.

Abstract

High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1\%, with no significant decline observed in other general capabilities.

Figures (5)

• Figure 1: The existing information extraction instructions significantly reduce the performance of LLMs in NLU tasks.
• Figure 2: Overview of the natural language understanding instruction synthesis framework. The framework consists of two parts: basic instruction synthesis and compound instruction synthesis. The compound instruction synthesis mainly comprises three strategies: guidelines synthesis, preference rules synthesis, and format variants synthesis.
• Figure 3: Prompt template for preference rule annotation.
• Figure 4: The source of the Hum dataset and the distribution of synthesis instructions.
• Figure 5: The performance of different large language models on the same compound instruction.