Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction

TL;DR

This work tackles the brittleness of Open Information Extraction under distributional drift by introducing ROBUST, a large-scale clique-based benchmark where sentences sharing the same knowledge meaning differ in syntactic form. It couples syntactically controlled paraphrase generation with a two-stage human annotation pipeline to create 1,272 robustness cliques (4,971 sentences, 16,191 extractions) and defines a clique-level robustness metric based on the worst-case per-sentence performance, ensuring compatibility with CaRB. Empirical results across six OpenIE baselines and ChatGPT reveal substantial robustness gaps on ROBUST (average F1 decreases around $18\%$), with SpanOIE being comparatively more robust and ChatGPT’s robustness still lagging behind; syntactic divergence correlates with increased performance variance. The findings emphasize the importance of robustness benchmarks for OpenIE and provide a practical framework and resources to advance distributionally robust information extraction systems.

Abstract

The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial measurement of robustness. In this paper, we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntactic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique that consists of sentences with structured knowledge of the same meaning but with different syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques. We perform experiments on typical models published in the last decade as well as a popular large language model, the results show that the existing successful models exhibit a frustrating degradation, with a maximum drop of 23.43 F1 score. Our resources and code are available at https://github.com/qijimrc/ROBUST.

Figures (9)

• Figure 1: Extraction results of OpenIE6 for three semantically equivalent sentences from CaRB and ROBUST. The proposed benchmark ROBUST computes the robustness score on a clique of sentences.
• Figure 2: An example of a robustness clique consisting of three sentences from ROBUST, where the sentences exhibit syntactic and expressive variants while preserving the same structured knowledge meaning. In contrast to conventional metrics, ROBUST measures the robustness score on a clique of all nodes.
• Figure 3: The average syntactic distances/similarity in each clique is calculated using HWS distance and Convolutional Tree Kernels, where the x-axis refers to the hierarchical discounting weights for two algorithms.
• Figure 4: The average syntactic distance/similarity over all cliques with the hierarchical discounting weights. Cliques containing only one point will be a line with a value of 0 or 1.
• Figure 5: (a) The distribution of the number of cliques with the variance of $F_1$ scores in each clique. (b) The variance of $F_1$ scores with the values HWS distance. (c) The variance of $F_1$ scores with the values of Convolutional Tree Kernel similarity. The both correlation values are divided into several intervals to avoid abnormal values.