Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Logic-Oriented Retriever Enhancement via Contrastive Learning

Wenxuan Zhang, Yuan-Hao Jiang, Changyong Qi, Rui Jia, Yonghe Wu

TL;DR

Knowledge-intensive retrieval with LLMs often suffers from reliance on surface similarity when queries involve complex logical relations. LORE introduces a post-pretrain contrastive learning framework that fine-tunes the query encoder using a fine-grained, RST-guided dataset (CoEnTrain) while freezing the document encoder, optimizing an InfoNCE-style objective $L(q)$ to emphasize logical structure, and uses a tiered distractor scheme with P, N1, N2. It provides a dataset construction method, a method for defining the positive probability $p_k$ for negatives, and validates improvements across HotpotQA, MS MARCO, and MuSiQue with cross-task generalization, all without external supervision. The approach preserves efficiency and open-source compatibility, offering a practical path to strengthen embedding models' foundational reasoning capabilities for retrieval-augmented generation.

Abstract

Large language models (LLMs) struggle in knowledge-intensive tasks, as retrievers often overfit to surface similarity and fail on queries involving complex logical relations. The capacity for logical analysis is inherent in model representations but remains underutilized in standard training. LORE (Logic ORiented Retriever Enhancement) introduces fine-grained contrastive learning to activate this latent capacity, guiding embeddings toward evidence aligned with logical structure rather than shallow similarity. LORE requires no external upervision, resources, or pre-retrieval analysis, remains index-compatible, and consistently improves retrieval utility and downstream generation while maintaining efficiency. The datasets and code are publicly available at https://github.com/mazehart/Lore-RAG.

Logic-Oriented Retriever Enhancement via Contrastive Learning

TL;DR

Knowledge-intensive retrieval with LLMs often suffers from reliance on surface similarity when queries involve complex logical relations. LORE introduces a post-pretrain contrastive learning framework that fine-tunes the query encoder using a fine-grained, RST-guided dataset (CoEnTrain) while freezing the document encoder, optimizing an InfoNCE-style objective to emphasize logical structure, and uses a tiered distractor scheme with P, N1, N2. It provides a dataset construction method, a method for defining the positive probability for negatives, and validates improvements across HotpotQA, MS MARCO, and MuSiQue with cross-task generalization, all without external supervision. The approach preserves efficiency and open-source compatibility, offering a practical path to strengthen embedding models' foundational reasoning capabilities for retrieval-augmented generation.

Abstract

Large language models (LLMs) struggle in knowledge-intensive tasks, as retrievers often overfit to surface similarity and fail on queries involving complex logical relations. The capacity for logical analysis is inherent in model representations but remains underutilized in standard training. LORE (Logic ORiented Retriever Enhancement) introduces fine-grained contrastive learning to activate this latent capacity, guiding embeddings toward evidence aligned with logical structure rather than shallow similarity. LORE requires no external upervision, resources, or pre-retrieval analysis, remains index-compatible, and consistently improves retrieval utility and downstream generation while maintaining efficiency. The datasets and code are publicly available at https://github.com/mazehart/Lore-RAG.
Paper Structure (10 sections, 5 equations, 3 figures, 1 table)

This paper contains 10 sections, 5 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Dataset Construction
  5. Contrastive Learning
  6. Experiments
  7. Experimental Setup
  8. Results and Analysis
  9. Limitations and Conclusion
  10. Acknowledgments

Figures (3)

  • Figure 1: Impact of complex logical expressions on embedding model performance. Results on MSMARCO dataset show significant performance degradation across different embedding models when queries are disturbed with complex logical expressions, highlighting the vulnerability of current embedding approaches to complex logical structures.
  • Figure 2: An illustration of Query Rewriting. The left side shows an original example with query, chunks, and labels, while the right side presents the corresponding modified example with enriched query, discourse relation, and distractor annotations.
  • Figure 3: Training and validation loss curves (upper) and chunk score evolution for different chunk categories (lower) during Qwen3-Embedding-0.6B fine-tuning.