Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction

TL;DR

HYDRE introduces a hybrid DSRE–ICL framework that first uses a distantly supervised model to generate high-recall candidate relations and then leverages a large language model as a judge with carefully constructed, dynamic exemplars to output final relation predictions. A three-stage exemplar retrieval process (candidate relation selection, bag selection, sentence selection) ensures clean, sentence-level exemplars despite noisy bag-level training data, and prompts are tailored to guide the LLM’s reasoning. The authors extend HYDRE to cross-lingual settings by curating a new multilingual Indic benchmark (Oriya, Santali, Manipuri, Tulu) and evaluating under English-only, translate-train, and translate-test transfer paradigms, reporting substantial gains over prior SoTA DSRE baselines (up to ~20 F1 in English and ~17 F1 on Indic languages). Ablation analyses show that retrieval components and the fusion of DSRE confidence with semantic similarity are critical for performance, and results suggest HYDRE is robust across monolingual and cross-lingual regimes with potential for further improvements in efficiency and domain coverage.

Abstract

Distantly Supervised Relation Extraction (DSRE) remains a long-standing challenge in NLP, where models must learn from noisy bag-level annotations while making sentence-level predictions. While existing state-of-the-art (SoTA) DSRE models rely on task-specific training, their integration with in-context learning (ICL) using large language models (LLMs) remains underexplored. A key challenge is that the LLM may not learn relation semantics correctly, due to noisy annotation. In response, we propose HYDRE -- HYbrid Distantly Supervised Relation Extraction framework. It first uses a trained DSRE model to identify the top-k candidate relations for a given test sentence, then uses a novel dynamic exemplar retrieval strategy that extracts reliable, sentence-level exemplars from training data, which are then provided in LLM prompt for outputting the final relation(s). We further extend HYDRE to cross-lingual settings for RE in low-resource languages. Using available English DSRE training data, we evaluate all methods on English as well as a newly curated benchmark covering four diverse low-resource Indic languages -- Oriya, Santali, Manipuri, and Tulu. HYDRE achieves up to 20 F1 point gains in English and, on average, 17 F1 points on Indic languages over prior SoTA DSRE models. Detailed ablations exhibit HYDRE's efficacy compared to other prompting strategies.

Figures (17)

• Figure 1: Hydre overview (shown for k=3): (a) Select top-k candidate labels using PARE confidence, (b) Select best bag for each candidate using combined semantic similarity and label confidence (aggregated over bag sentences), and (c) Select best sentence per bag using aggregate (over bag labels) confidence scoring.
• Figure 2: Zero-shot prompting vs full bag ICL vs reduced bag ICL for GPT-4o on English NYT-10m
• Figure 3: Confusion analysis of Hydre vs the baselines (aggregated over all relations)
• Figure 4: Example 4: Hydre correctly predicts the "Religion" label while 0-shot confuses it with "Ethnicity".
• Figure 5: Example 5: Hydre correctly predicts the "Advisors" label while 0-shot misses it.