Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Best of Both Worlds: A Pliable and Generalizable Neuro-Symbolic Approach for Relation Classification

Robert Vacareanu, Fahmida Alam, Md Asiful Islam, Haris Riaz, Mihai Surdeanu

TL;DR

This paper introduces a novel neuro-symbolic architecture for relation classification that combines rule-based methods with contemporary deep learning techniques and shows that the proposed method outperforms previous state-of-the-art models in three out of four settings, despite not seeing any human-annotated training data.

Abstract

This paper introduces a novel neuro-symbolic architecture for relation classification (RC) that combines rule-based methods with contemporary deep learning techniques. This approach capitalizes on the strengths of both paradigms: the adaptability of rule-based systems and the generalization power of neural networks. Our architecture consists of two components: a declarative rule-based model for transparent classification and a neural component to enhance rule generalizability through semantic text matching. Notably, our semantic matcher is trained in an unsupervised domain-agnostic way, solely with synthetic data. Further, these components are loosely coupled, allowing for rule modifications without retraining the semantic matcher. In our evaluation, we focused on two few-shot relation classification datasets: Few-Shot TACRED and a Few-Shot version of NYT29. We show that our proposed method outperforms previous state-of-the-art models in three out of four settings, despite not seeing any human-annotated training data. Further, we show that our approach remains modular and pliable, i.e., the corresponding rules can be locally modified to improve the overall model. Human interventions to the rules for the TACRED relation \texttt{org:parents} boost the performance on that relation by as much as 26\% relative improvement, without negatively impacting the other relations, and without retraining the semantic matching component.

Best of Both Worlds: A Pliable and Generalizable Neuro-Symbolic Approach for Relation Classification

TL;DR

This paper introduces a novel neuro-symbolic architecture for relation classification that combines rule-based methods with contemporary deep learning techniques and shows that the proposed method outperforms previous state-of-the-art models in three out of four settings, despite not seeing any human-annotated training data.

Abstract

This paper introduces a novel neuro-symbolic architecture for relation classification (RC) that combines rule-based methods with contemporary deep learning techniques. This approach capitalizes on the strengths of both paradigms: the adaptability of rule-based systems and the generalization power of neural networks. Our architecture consists of two components: a declarative rule-based model for transparent classification and a neural component to enhance rule generalizability through semantic text matching. Notably, our semantic matcher is trained in an unsupervised domain-agnostic way, solely with synthetic data. Further, these components are loosely coupled, allowing for rule modifications without retraining the semantic matcher. In our evaluation, we focused on two few-shot relation classification datasets: Few-Shot TACRED and a Few-Shot version of NYT29. We show that our proposed method outperforms previous state-of-the-art models in three out of four settings, despite not seeing any human-annotated training data. Further, we show that our approach remains modular and pliable, i.e., the corresponding rules can be locally modified to improve the overall model. Human interventions to the rules for the TACRED relation \texttt{org:parents} boost the performance on that relation by as much as 26\% relative improvement, without negatively impacting the other relations, and without retraining the semantic matching component.
Paper Structure (25 sections, 3 figures, 8 tables)

This paper contains 25 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Rule-based Approaches
  4. Explainable Deep Learning
  5. Prompting and In-context Learning
  6. Proposed Method
  7. Strict Rule Matching Component
  8. Rule Generation
  9. Semantic Rule Matching Component
  10. Training Dataset
  11. Training the Semantic Rule Matching
  12. Experiments
  13. Experimental Setup
  14. Baselines
  15. Main Results
  16. ...and 10 more sections

Figures (3)

  • Figure 1: An example of the type of rules we use in our proposed method, together with three sentences. The rule captures the org:founder relation with a syntactic pattern anchored by the predicate founded that has a person named entity as its subject and an organization as the direct object. By itself, the rule matches the first sentence, but it does not match the other two. When coupled with our semantic matching component, the rule matches the first two sentences.
  • Figure 2: To create a rule from a sentence, the process involves: (a) parsing the sentence to extract its syntactic dependency tree, (b) identifying the shortest path connecting two entity mentions within this tree, and (c) constructing a rule based on the syntactic dependencies, associated words, and named entity labels found along this path. For example, the rule shown operates as follows: it requires a per (person) label connected to the word 'founder' via a nominal subject dependency, and 'founder' in turn linked to a org (organization) label through an nmod_of dependency.
  • Figure 3: In our training for the Semantic Rule Matcher (SRM), we encode both rules and sentences, followed by calculating cosine similarity between each pair. The goal is to maximize similarity for matching pairs (diagonal of the matrix) and minimize it for non-matching pairs (off-diagonal elements).