AmbigNLG: Addressing Task Ambiguity in Instruction for NLG

TL;DR

An ambiguity taxonomy is proposed that categorizes different types of instruction ambiguities and refines initial instructions with clearer specifications and confirms the effectiveness of the method in practical settings involving interactive ambiguity mitigation with users, underscoring the benefits of leveraging LLMs for interactive clarification.

Abstract

We introduce AmbigNLG, a novel task designed to tackle the challenge of task ambiguity in instructions for Natural Language Generation (NLG). Ambiguous instructions often impede the performance of Large Language Models (LLMs), especially in complex NLG tasks. To tackle this issue, we propose an ambiguity taxonomy that categorizes different types of instruction ambiguities and refines initial instructions with clearer specifications. Accompanying this task, we present AmbigSNI-NLG, a dataset comprising 2,500 instances annotated to facilitate research in AmbigNLG. Through comprehensive experiments with state-of-the-art LLMs, we demonstrate that our method significantly enhances the alignment of generated text with user expectations, achieving up to a 15.02-point increase in ROUGE scores. Our findings highlight the critical importance of addressing task ambiguity to fully harness the capabilities of LLMs in NLG tasks. Furthermore, we confirm the effectiveness of our method in practical settings involving interactive ambiguity mitigation with users, underscoring the benefits of leveraging LLMs for interactive clarification.

Figures (7)

• Figure 1: Overview of our mitigation approach for the AmbigNLG task. We address task ambiguity by incorporating additional instructions into the initial instruction, thereby refining the task definition and improving the alignment of generated outputs with user expectations.
• Figure 2: Dataset creation process. The process includes curating high-quality manual annotations, generating additional instruction candidates, and validating these candidates to ensure clarity and utility.
• Figure 3: Distributions of the dataset. The upper bar graph displays the number of instances per task, while the lower line graph shows the proportion of instances assigned to ambiguous categories for each task.
• Figure 4: Mitigation results for each taxonomy.
• Figure 5: Mitigation results across the top-6 most frequent tasks in AmbigSNI$_{\texttt{NLG}}$. The figure demonstrates that ambiguity mitigation consistently enhances performance across different NLG tasks, as indicated by the ROUGE-L score improvements.