Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

Juri Opitz

TL;DR

Smatch++ tackles the inconsistent and opaque evaluation of semantic graphs by decomposing Smatch into three modules: pre-processing, alignment, and scoring. It standardizes preprocessing (including reification-based standardization and duplicate-triple removal), enables optimal or bounded alignment via ILP with a lossless graph-compression strategy, and extends scoring to fine-grained sub-graph aspects with macro- and confidence-interval reporting. The approach demonstrates that optimal alignment improves evaluation safety and that compression yields substantial speedups, while reification can enhance fairness in comparisons. The framework provides a practical, extensible path for fair, reproducible MR evaluation and richer diagnostics for parsing systems. Overall, Smatch++ advances standardized, extensible graph evaluation with quantified uncertainty and deeper semantic insight.

Abstract

The Smatch metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems. However, we observe some issues in the metric that jeopardize meaningful evaluation. E.g., opaque pre-processing choices can affect results, and current graph-alignment solvers do not provide us with upper-bounds. Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore, adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity) are spread out, and lack a unifying framework. For better inspection, we divide the metric into three modules: pre-processing, alignment, and scoring. Examining each module, we specify its goals and diagnose potential issues, for which we discuss and test mitigation strategies. For pre-processing, we show how to fully conform to annotation guidelines that allow structurally deviating but valid graphs. For safer and enhanced alignment, we show the feasibility of optimal alignment in a standard evaluation setup, and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency. For improved scoring, we propose standardized and extended metric calculation of fine-grained sub-graph meaning aspects. Our code is available at https://github.com/flipz357/smatchpp

SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

TL;DR

Smatch++ tackles the inconsistent and opaque evaluation of semantic graphs by decomposing Smatch into three modules: pre-processing, alignment, and scoring. It standardizes preprocessing (including reification-based standardization and duplicate-triple removal), enables optimal or bounded alignment via ILP with a lossless graph-compression strategy, and extends scoring to fine-grained sub-graph aspects with macro- and confidence-interval reporting. The approach demonstrates that optimal alignment improves evaluation safety and that compression yields substantial speedups, while reification can enhance fairness in comparisons. The framework provides a practical, extensible path for fair, reproducible MR evaluation and richer diagnostics for parsing systems. Overall, Smatch++ advances standardized, extensible graph evaluation with quantified uncertainty and deeper semantic insight.

Abstract

The Smatch metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems. However, we observe some issues in the metric that jeopardize meaningful evaluation. E.g., opaque pre-processing choices can affect results, and current graph-alignment solvers do not provide us with upper-bounds. Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore, adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity) are spread out, and lack a unifying framework. For better inspection, we divide the metric into three modules: pre-processing, alignment, and scoring. Examining each module, we specify its goals and diagnose potential issues, for which we discuss and test mitigation strategies. For pre-processing, we show how to fully conform to annotation guidelines that allow structurally deviating but valid graphs. For safer and enhanced alignment, we show the feasibility of optimal alignment in a standard evaluation setup, and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency. For improved scoring, we propose standardized and extended metric calculation of fine-grained sub-graph meaning aspects. Our code is available at https://github.com/flipz357/smatchpp
Paper Structure (47 sections, 6 equations, 6 figures, 4 tables)

This paper contains 47 sections, 6 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Paper structure and contributions
  3. Related work
  4. Metric standardization
  5. MR metrics
  6. Smatch: Overview and generalization
  7. Preliminary I: MR graph
  8. Preliminary II: Smatch
  9. Generalizing Smatch
  10. A modular view on Smatch
  11. Module I: Pre-processing
  12. Module goal and current implementation
  13. Two structures, one meaning: reification
  14. Empirical assessment of effect
  15. Discussion
  16. ...and 32 more sections

Figures (6)

  • Figure 1: A serialized MR string is read into a graph.
  • Figure 2: Outline of location-reification.
  • Figure 3: Sketch of search space (top) and hill-climber run (bottom). Every hill-climber step constitutes an improved lower bound, but we cannot obtain a tight upper-bound (an accessible trivial upper-bound is the amount of triples in the smaller of two graphs: 3).
  • Figure 4: Named Entitiy (NE) sub-graph extraction with Smatch vs. Smatch++
  • Figure 5: Temporal sub-graph extraction with Smatch++ for an MR capturing "Baked goods flourished at the end of the fourth century".
  • ...and 1 more figures