Transformers in the Service of Description Logic-based Contexts
Angelos Poulis, Eleni Tsalapati, Manolis Koubarakis
TL;DR
This work introduces DELTA$_D$, a large, expressive NL dataset for entailment over $ \mathcal{ALCQ}$ knowledge bases, generated via probabilistic context-free grammars to span multiple inference depths $\mathcal{D}$ and linguistic complexity levels $\mathcal{L}$. It demonstrates that a fine-tuned DeBERTaV3-large model (DELTA$_M$) generalizes well across depths and complexities, achieving near-perfect accuracy on the diverse test sets, while also revealing limits in certain symbolic/logical distributions. The authors further evaluate GPT-3.5 and GPT-4 under few-shot settings, showing GPT-4 can reach substantial accuracy with as few as 9 shots, though performance degrades at greater reasoning depths. They provide extensive analyses, including zero-shot robustness across altered distributions, symbolic data tests, and a real-world fuel-cell use case, and openly release code and data. Overall, the paper argues that transformer-based models can perform DL-style reasoning on complex, expressive contexts, highlighting both strong capabilities and areas for methodological advancement and evaluation.
Abstract
Recent advancements in transformer-based models have initiated research interests in investigating their ability to learn to perform reasoning tasks. However, most of the contexts used for this purpose are in practice very simple: generated from short (fragments of) first-order logic sentences with only a few logical operators and quantifiers. In this work, we construct the natural language dataset, DELTA$_D$, using the description logic language $\mathcal{ALCQ}$. DELTA$_D$ contains 384K examples, and increases in two dimensions: i) reasoning depth, and ii) linguistic complexity. In this way, we systematically investigate the reasoning ability of a supervised fine-tuned DeBERTa-based model and of two large language models (GPT-3.5, GPT-4) with few-shot prompting. Our results demonstrate that the DeBERTa-based model can master the reasoning task and that the performance of GPTs can improve significantly even when a small number of samples is provided (9 shots). We open-source our code and datasets.