Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification

TL;DR

The paper tackles zero-shot political relation classification within evolving ontologies by leveraging annotation codebooks. It introduces ZSP, a mode-aware NLI framework that decomposes classification into context, mode, and disambiguation, and compares it with ChatGPT-based prompts. Empirical results show ZSP often outperforms dictionary-based approaches and remains competitive with supervised models, while GPT-4 improves generalization over GPT-3.5 but may still lag behind ZSP in fine-grained tasks. The work demonstrates that integrating domain knowledge with principled NLI and efficient tree-query reasoning can reduce annotation costs, enable rapid ontology updates, and scale to diverse political-event datasets.

Abstract

Is it possible accurately classify political relations within evolving event ontologies without extensive annotations? This study investigates zero-shot learning methods that use expert knowledge from existing annotation codebook, and evaluates the performance of advanced ChatGPT (GPT-3.5/4) and a natural language inference (NLI)-based model called ZSP. ChatGPT uses codebook's labeled summaries as prompts, whereas ZSP breaks down the classification task into context, event mode, and class disambiguation to refine task-specific hypotheses. This decomposition enhances interpretability, efficiency, and adaptability to schema changes. The experiments reveal ChatGPT's strengths and limitations, and crucially show ZSP's outperformance of dictionary-based methods and its competitive edge over some supervised models. These findings affirm the value of ZSP for validating event records and advancing ontology development. Our study underscores the efficacy of leveraging transfer learning and existing domain expertise to enhance research efficiency and scalability.

Figures (10)

• Figure 1: Event coding illustration: How event modes affect Rootcode and Quadcode labeling for sentences involving identical entities.
• Figure 2: Two zero-shot approaches for classifying relation labels (Rootcode and Quadcode) in a source - target pair. ChatGPT employs prompts designed from the codebook's label summaries, while ZSP utilizes a pretrained NLI model and a tree-query system. Hypotheses and class disambiguation rules are derived from the codebook and enhanced with mode considerations (e.g., Past, Future). The tree-query framework reduces query time and improves precision by filtering candidates, determining modes, and eliminating ambiguity.
• Figure 3: Performance vs. varying sized training datasets.
• Figure 4: Examples of templates in the A/W's original ontology wikievent.
• Figure 5: Extending PLV-Quadcode to Rootcode level.