TabVer: Tabular Fact Verification with Natural Logic

TL;DR

This work proposes a set-theoretic interpretation of numerals and arithmetic functions in the context of natural logic, enabling the integration of arithmetic expressions in deterministic proofs by leveraging large language models to generate arithmetic expressions.

Abstract

Fact verification on tabular evidence incentivises the use of symbolic reasoning models where a logical form is constructed (e.g. a LISP-style program), providing greater verifiability than fully neural approaches. However, these systems typically rely on well-formed tables, restricting their use in many scenarios. An emerging symbolic reasoning paradigm for textual evidence focuses on natural logic inference, which constructs proofs by modelling set-theoretic relations between a claim and its evidence in natural language. This approach provides flexibility and transparency but is less compatible with tabular evidence since the relations do not extend to arithmetic functions. We propose a set-theoretic interpretation of numerals and arithmetic functions in the context of natural logic, enabling the integration of arithmetic expressions in deterministic proofs. We leverage large language models to generate arithmetic expressions by generating questions about salient parts of a claim which are answered by executing appropriate functions on tables. In a few-shot setting on FEVEROUS, we achieve an accuracy of 71.4, outperforming both fully neural and symbolic reasoning models by 3.4 points. When evaluated on TabFact without any further training, our method remains competitive with an accuracy lead of 0.5 points.

Figures (7)

• Figure 1: High-level illustration of TabVer. TabVer proposes a set-theoretic view on numerals and arithmetic functions, which is integrated into natural logic proofs as arithmetic comparisons between claim and answers to questions (ArithExps), resulting in deterministic inference (left). To generate ArithExps, TabVer asks questions about salient parts $c_i$ of a claim (middle). The questions are answered using tabular evidence $E$, by generating a rationale and a set-theoretic compatible representation of required computations (right).
• Figure 2: The finite state automaton (DFA), following natural logic inference angeli-manning-2014-naturalli. The transitions in the DFA denote NatOps and the states the veracity labels. The final state on which the the proof terminates determines the overall veracity.
• Figure 3: A set-theoretic view of the relationship between claim spans $c_i$ and numerical expressions in the evidence $e_i$ when following an upper-bounded interpretation of numerals.
• Figure 4: Tabular question answering via a rationale that produces the answer $a_i$ via an ArithExp. An LLM jointly extracts relevant information from table cells and executes appropriate functions. The generation is constrained to permissible functions and to numbers that appear in the evidence to alleviate hallucinations.
• Figure 5: Illustrating claim decomposition and verdict aggregation. Further, claim decomposition partially addresses the issue of producing less informative predictions by constraining natural logic to left-to-right execution.