Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives

Aarush Sinha, Pavan Kumar S, Roshan Balaji, Nirav Pravinbhai Bhatt

TL;DR

BiCA introduces a citation-aware hard-negative mining strategy for biomedical dense retrieval by constructing 2-hop PubMed citation neighborhoods and performing diverse semantic traversals to mine challenging negatives. The authors train two compact dense retrievers (BiCA_small and BiCA_Base) with MNR loss on a 20k-pair dataset and achieve state-of-the-art zero-shot results on BEIR and strong performance on LoTTE, with BiCA_small offering substantial latency advantages. The approach emphasizes data efficiency, showing clear gains with limited fine-tuning steps and highlighting the potential of document-link structures for domain adaptation. Limitations include scalability and reliance on PubMed, suggesting future expansion to other knowledge sources.

Abstract

Hard negatives are essential for training effective retrieval models. Hard-negative mining typically relies on ranking documents using cross-encoders or static embedding models based on similarity metrics such as cosine distance. Hard negative mining becomes challenging for biomedical and scientific domains due to the difficulty in distinguishing between source and hard negative documents. However, referenced documents naturally share contextual relevance with the source document but are not duplicates, making them well-suited as hard negatives. In this work, we propose BiCA: Biomedical Dense Retrieval with Citation-Aware Hard Negatives, an approach for hard-negative mining by utilizing citation links in 20,000 PubMed articles for improving a domain-specific small dense retriever. We fine-tune the GTE_small and GTE_Base models using these citation-informed negatives and observe consistent improvements in zero-shot dense retrieval using nDCG@10 for both in-domain and out-of-domain tasks on BEIR and outperform baselines on long-tailed topics in LoTTE using Success@5. Our findings highlight the potential of leveraging document link structure to generate highly informative negatives, enabling state-of-the-art performance with minimal fine-tuning and demonstrating a path towards highly data-efficient domain adaptation.

BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives

TL;DR

BiCA introduces a citation-aware hard-negative mining strategy for biomedical dense retrieval by constructing 2-hop PubMed citation neighborhoods and performing diverse semantic traversals to mine challenging negatives. The authors train two compact dense retrievers (BiCA_small and BiCA_Base) with MNR loss on a 20k-pair dataset and achieve state-of-the-art zero-shot results on BEIR and strong performance on LoTTE, with BiCA_small offering substantial latency advantages. The approach emphasizes data efficiency, showing clear gains with limited fine-tuning steps and highlighting the potential of document-link structures for domain adaptation. Limitations include scalability and reliance on PubMed, suggesting future expansion to other knowledge sources.

Abstract

Hard negatives are essential for training effective retrieval models. Hard-negative mining typically relies on ranking documents using cross-encoders or static embedding models based on similarity metrics such as cosine distance. Hard negative mining becomes challenging for biomedical and scientific domains due to the difficulty in distinguishing between source and hard negative documents. However, referenced documents naturally share contextual relevance with the source document but are not duplicates, making them well-suited as hard negatives. In this work, we propose BiCA: Biomedical Dense Retrieval with Citation-Aware Hard Negatives, an approach for hard-negative mining by utilizing citation links in 20,000 PubMed articles for improving a domain-specific small dense retriever. We fine-tune the GTE_small and GTE_Base models using these citation-informed negatives and observe consistent improvements in zero-shot dense retrieval using nDCG@10 for both in-domain and out-of-domain tasks on BEIR and outperform baselines on long-tailed topics in LoTTE using Success@5. Our findings highlight the potential of leveraging document link structure to generate highly informative negatives, enabling state-of-the-art performance with minimal fine-tuning and demonstrating a path towards highly data-efficient domain adaptation.

Paper Structure

This paper contains 25 sections, 1 equation, 2 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Related Work
  4. Biomedical Information Retrieval
  5. Biomedical Language Models
  6. Methodology
  7. Data Curation: 2-Hop Citation Neighborhood Construction
  8. Hard-Negative Mining via Diverse Semantic Traversal
  9. Experiments
  10. Fine-Tuning
  11. Evaluation
  12. BEIR
  13. LOTTE
  14. Latency
  15. Effect of Traversal Parameters
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Our four-stage data generation and training pipeline. Stage 1: A query is synthetically generated from a positive document's abstract using a T5 model. Stage 2: A 2-hop citation neighborhood is constructed by retrieving papers cited by the positive document (1-hop) and papers cited by them (2-hop) via the PubMed API. Stage 3: Hard negatives are mined via semantic graph traversal. First, similarities are computed between the query and 1-hop documents. Second, a dense, pairwise similarity graph is built for all 1-hop and 2-hop documents. Third, a 5-step greedy traversal is initiated from the 1-hop document most similar to the query, creating a path of five hard negatives. Stage 4: The resulting (Query, Positive Document, Hard Negatives) triplet is used to fine-tune the GTE model using the multiple negative ranking loss.
  • Figure 2: (a) Corpus embedding distribution comparison and (b) validation nDCG@10 across training steps.