Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings

Xiaoyu Liang, Yuchen Peng, Jiale Luo, Wenhao Wang, Haoji Hu, Xincheng Zhou

TL;DR

This work addresses the difficulty of learning accurate, domain-specific embeddings with LLMs in vertical domains where domain knowledge is sparse. It introduces Learn Before Represent (LBR), a two-stage framework that first injects knowledge via Information Bottleneck-constrained Generative Learning (Stage 1) to compress semantics into bottleneck tokens while preserving causal attention, then applies Generative-Refined Contrastive Learning (Stage 2) to align representations using those compressed tokens. The approach unifies generative and contrastive objectives, mitigating representation collapse and architectural inconsistencies, and it demonstrates significant improvements across medicine, chemistry, and code retrieval tasks compared to strong baselines. The results establish a new paradigm for robust, domain-specific embeddings, balancing knowledge acquisition with semantic alignment for vertical-domain applications.

Abstract

Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains.

Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings

TL;DR

This work addresses the difficulty of learning accurate, domain-specific embeddings with LLMs in vertical domains where domain knowledge is sparse. It introduces Learn Before Represent (LBR), a two-stage framework that first injects knowledge via Information Bottleneck-constrained Generative Learning (Stage 1) to compress semantics into bottleneck tokens while preserving causal attention, then applies Generative-Refined Contrastive Learning (Stage 2) to align representations using those compressed tokens. The approach unifies generative and contrastive objectives, mitigating representation collapse and architectural inconsistencies, and it demonstrates significant improvements across medicine, chemistry, and code retrieval tasks compared to strong baselines. The results establish a new paradigm for robust, domain-specific embeddings, balancing knowledge acquisition with semantic alignment for vertical-domain applications.

Abstract

Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains.
Paper Structure (33 sections, 4 equations, 3 figures, 6 tables)

This paper contains 33 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based Text Representation Learning
  4. Preserving Causality: Embedding without Forgetting Generation
  5. Representation Enhancement via Information Bottleneck
  6. Method
  7. Preliminaries: The Information Bottleneck Principle
  8. Stage 1: IB-Constrained Generative Learning
  9. Minimizing $\colorbox{green!10}{$I(X; Z)$}$ via Information Cut-off.
  10. Maximizing $\colorbox{yellow!10}{$I(Z; Y)$}$ via Generative Loss.
  11. Information-Theoretic Interpretation.
  12. Stage 2: Generative-Refined Contrastive Learning
  13. Experiments
  14. Experimental Settings
  15. Datasets.
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Motivation of the Learn Before Represent (LBR) framework.$(Top)$ Only Contrastive (LLM+CL) methods focus on semantic alignment but fail on domain-specific entities (e.g., matching "Acetylsalicylic" to "Aspirin") due to a lack of internal knowledge. $(Bottom)$ Only Generative (LLM+GL) methods acquire knowledge via Next-Token Prediction but suffer from representation collapse. We addresses these challenges through a $\textbf{Learn \ Before \ Represent}$ framework.
  • Figure 2: Overview of the LBR framework. In Stage 1, the model performs generative learning under an Information Bottleneck constraint, $compressing$ input semantics into bottleneck tokens and $predicting$ the target solely from these tokens. In Stage 2, the bottleneck tokens serve as embeddings, better $representing$ the semantics through contrastive learning. Both stages operate under a unified causal attention mechanism to maximize knowledge transfer.
  • Figure 3: Impact of GL data allocation. We fix the total training budget at 100k samples and vary the GL data ratio ($r_{\text{learn}}$) from 0% to 100% in Stage 1, where $r_{\text{learn}}=0\%$ represents pure contrastive learning and $r_{\text{learn}}=100\%$ represents pure generative learning. Both our IB-constrained method and standard GL are evaluated under identical data allocation settings.