Verifiable Natural Language to Linear Temporal Logic Translation: A Benchmark Dataset and Evaluation Suite

TL;DR

The paper introduces VLTL-Bench, a verifiable NL-to-LTL benchmark that emphasizes grounding and verification by exposing lifting, grounding, translation, and verification steps across four diverse scenarios. It provides grounded LTL formulas, AP dictionaries, and execution traces to support intermediate-grounding evaluation and end-to-end benchmarking. Four metrics enable granular assessment of each NL-to-LTL subtask and the overall pipeline, including a trace-based verification component. Experimental results reveal that current NL-to-TL systems struggle with grounding and that high lifted-translation accuracy does not guarantee executable, verifiable specifications in real-world state spaces.

Abstract

Empirical evaluation of state-of-the-art natural-language (NL) to temporal-logic (TL) translation systems reveals near-perfect performance on existing benchmarks. However, current studies measure only the accuracy of the translation of NL logic into formal TL, ignoring a system's capacity to ground atomic propositions into new scenarios or environments. This is a critical feature, necessary for the verification of resulting formulas in a concrete state space. Consequently, most NL-to-TL translation frameworks propose their own bespoke dataset in which the correct grounding is known a-priori, inflating performance metrics and neglecting the need for extensible, domain-general systems. In this paper, we introduce the Verifiable Linear Temporal Logic Benchmark ( VLTL-Bench), a unifying benchmark that measures verification and verifiability of automated NL-to-LTL translation. The dataset consists of four unique state spaces and thousands of diverse natural language specifications and corresponding formal specifications in temporal logic. Moreover, the benchmark contains sample traces to validate the temporal logic expressions. While the benchmark directly supports end-to-end evaluation, we observe that many frameworks decompose the process into i) lifting, ii) grounding, iii) translation, and iv) verification. The benchmark provides ground truths after each of these steps to enable researches to improve and evaluate different substeps of the overall problem. To encourage methodologically sound advances in verifiable NL-to-LTL translation approaches, we release VLTL-Bench here: https://www.kaggle.com/datasets/dubascudes/vltl bench.

Figures (6)

• Figure 1: Overview of our dataset synthesis and evaluation framework for NL-to-LTL translation. The framework used a configuration file to define concrete and unique scenarios. The data synthesis generates the NL and TL pairs with associated traces for verification while providing ground truth results for intermediate components.
• Figure 2: Overview of an isolated evaluation of each individual component. Lifting accuracy measures accuracy of predicted natural language AP spans, grounding accuracy measures the performance on mapping AP spans to world state conditions, translation accuracy measures the performance on NL-LTL translation on the token-level, and verification accuracy is an approach to measuring whether a grounded LTL expression holds on a trace.
• Figure 3: Warehouse Scenario Configuration file
• Figure 4: Traffic Light Scenario Configuration file
• Figure 5: Search and Rescue Scenario Configuration file