Handling Ontology Gaps in Semantic Parsing

TL;DR

The paper tackles the risk of hallucinations in neural semantic parsing under a closed-world assumption, where queries requiring unseen ontology symbols can yield wrong or unsafe answers. It introduces the Hallucination Simulation Framework (HSF) to programmatically induce and study NSP hallucinations, and a Hallucination Detection Model (HDM) that leverages multiple signals—Confidence Score, Monte Carlo Dropout, and model activations—to detect and prevent hallucinated MRLs at inference time. A Hallucination Detection Dataset (HDD) is constructed by partitioning ontology into known and unknown symbol sets and incorporating out-of-domain and zero-shot cases, enabling robust evaluation across in-ontology errors, ontology gaps, and OOD inputs. Experiments on the KQA-PRO dataset show that combining Activations with CS (and optionally MCD) within the HDM yields the largest Macro F1 improvements, notably up to ~21% for ontology gaps and ~24% for OOD detection, with modest gains for NSP errors. The work provides a first systematic treatment of ontology gaps in closed-ontology NSP and demonstrates a practical, low-latency strategy to enhance the trustworthiness of NSP-based QA systems.

Abstract

The majority of Neural Semantic Parsing (NSP) models are developed with the assumption that there are no concepts outside the ones such models can represent with their target symbols (closed-world assumption). This assumption leads to generate hallucinated outputs rather than admitting their lack of knowledge. Hallucinations can lead to wrong or potentially offensive responses to users. Hence, a mechanism to prevent this behavior is crucial to build trusted NSP-based Question Answering agents. To that end, we propose the Hallucination Simulation Framework (HSF), a general setting for stimulating and analyzing NSP model hallucinations. The framework can be applied to any NSP task with a closed-ontology. Using the proposed framework and KQA Pro as the benchmark dataset, we assess state-of-the-art techniques for hallucination detection. We then present a novel hallucination detection strategy that exploits the computational graph of the NSP model to detect the NSP hallucinations in the presence of ontology gaps, out-of-domain utterances, and to recognize NSP errors, improving the F1-Score respectively by ~21, ~24% and ~1%. This is the first work in closed-ontology NSP that addresses the problem of recognizing ontology gaps. We release our code and checkpoints at https://github.com/amazon-science/handling-ontology-gaps-in-semantic-parsing.

Figures (7)

• Figure 1: The proposed pipeline: (1) the NSP model (KQA-PRO Bart model) receives the question from the user and produces the corresponding MRL; (2) the Hallucination Detection Model extracts features from the NSP model and decides whether to deliver the MRL to the user or not.
• Figure 2: We show the output our NSP model trained without a symbol for the concept of "cause of death". Given a question that requires this symbol, the model produces a wrong but executable MRL leading to a wrong answer served to its user.
• Figure 3: Construction of the Hallucination Detection Dataset (HDD). The first row represents the dataset used to train and test the NSP model, containing only $\mathcal{O}_{\text{known\_symbols}}$. To construct the $\mathcal{O}_{\text{known\_symbols}}$ portion of the HDD while avoiding overfitting of the hallucination detection strategies, we sourced sentences only from the validation and test splits of the NSP dataset as explained in Section \ref{['sec:in-ontology-bias']}.
• Figure 4: Overlap between the distributions of correct predictions, out-of-ontology, NSP errors, and OOD w.r.t. Confidence Score (CS). The model is overconfident over wrong predictions, hence the CS is not sufficient to separate good and Hallucinated MRLs. Specifically, the CS struggles to distinguish between NSP Errors and correct predictions (i.e., both types of MRLs that contains only $\mathcal{O}_{\text{known\_symbols}}$).
• Figure 5: In this plot on the y-axis the percentage of remaining error ($\downarrow$ is better) and on the x-axis we distinguish between the various hallucinated MRLs that omit (right plot) or add (left plot) incorrect ontology symbols with respect to the ground truth. Residual error compares two systems: Threshold CS and HDM with Activations and CS.