Risks and NLP Design: A Case Study on Procedural Document QA

TL;DR

The paper introduces the Risk-Aware Design Questionnaire (RADQ) to specialize risk analysis for Procedural Document QA (ProcDocQA) and applies it to a recipe-domain case study. It shows that zero-shot GPT-3 can match human answers on recipes, yet deeper multi-output evaluation reveals concrete risks such as output instability, hallucination, and unsafe recommendations that require design mitigations. Through a multi-decoding analysis, the authors identify actionable error categories and propose design strategies, including uncertainty visualization and source verification, to reduce risk. A post-study RADQ update further integrates risk-communication insights, outlining a path toward safer, user-aware deployment of procedural QA systems. The work advocates documenting error analyses across multiple outputs and employing interdisciplinary methods to align NLP systems with user capabilities and safety requirements.

Abstract

As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users--and concrete strategies to mitigate them--will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in "zero-shot" mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.

Figures (18)

• Figure 1: Dimensions characterizing procedural documents that can assist with estimating potential harms: risk of harm to the user or environment and the expertise required for the user to successfully complete the procedure.
• Figure 2: Annotators judged answers for correctness and could state their uncertainty about the answer correctness. Correct answers were judged for how they could be improved. Perfect answers required no change. Responses in i. and ii. were judged by experts, and iii. and iv. had crowdworker judges. GPT-3 generated responses in i. and iii. Human-written answers were judged in ii. and iv. Inter-annotator agreement about answer correctness was low for each group (Krippendorf's $\alpha<.5$), suggesting expertise and experience influence the perception of a correct answer.
• Figure 3: Output instability. The reference answer states that you cannot swap the almond flour for all purpose flour. Decoding 1 agrees, while decodings 2--4 state the opposite. All decodings suggest different usage of almond extract. Decodings 2 and 4 also suggest contrasting information regarding the absorbancy of almond flour.
• Figure 4: Leading question agreement, hallucination, and recommendations. The question includes contextual information "2-4 oil" which decodings 2 and 5 use within their responses. Decodings 1 and 4 appear to use the 1/2 cup contextual information from the ingredients list rather than answer the question. Decodings 2 and 4 recommend different recipe URLs that do not exist.
• Figure 5: Hallucination, language style behaviors. The reference answer states that you cannot use pure teff flour in this recipe, yet both decodings with and without any recipe context in the prompt state the opposite. However, the recipe only optionally uses teff at $\approx 25\%$ of the total flour content by weight, suggesting that you can't use all teff, regardless of any knowledge about the properties of teff (a dense gluten-free grain).