Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domain

Yidan Sun, Mengying Zhu, Feiyue Chen, Yangyang Wu, Xiaolei Dan, Mengyuan Yang, Xiaolin Zheng, Shenglin Ben

TL;DR

The paper tackles the isotropy of LLM embeddings that impairs domain-specific terminology discrimination in legal and financial contexts. It introduces TermGPT, a terminology-aware fine-tuning framework that couples sentence-graph based data augmentation with multi-level contrastive learning (sentence- and token-level) to balance global context and fine-grained term distinctions. It contributes a specialized financial terminology dataset and demonstrates that TermGPT yields notable improvements over domain-pretrained and contrastive baselines on terminology QCA and QA tasks, with robust cross-domain performance. The work has practical impact for high-stakes NLP tasks in regulated domains and points to future work on adversarial robustness.

Abstract

Large language models (LLMs) have demonstrated impressive performance in text generation tasks; however, their embedding spaces often suffer from the isotropy problem, resulting in poor discrimination of domain-specific terminology, particularly in legal and financial contexts. This weakness in terminology-level representation can severely hinder downstream tasks such as legal judgment prediction or financial risk analysis, where subtle semantic distinctions are critical. To address this problem, we propose TermGPT, a multi-level contrastive fine-tuning framework designed for terminology adaptation. We first construct a sentence graph to capture semantic and structural relations, and generate semantically consistent yet discriminative positive and negative samples based on contextual and topological cues. We then devise a multi-level contrastive learning approach at both the sentence and token levels, enhancing global contextual understanding and fine-grained terminology discrimination. To support robust evaluation, we construct the first financial terminology dataset derived from official regulatory documents. Experiments show that TermGPT outperforms existing baselines in term discrimination tasks within the finance and legal domains.

TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domain

TL;DR

The paper tackles the isotropy of LLM embeddings that impairs domain-specific terminology discrimination in legal and financial contexts. It introduces TermGPT, a terminology-aware fine-tuning framework that couples sentence-graph based data augmentation with multi-level contrastive learning (sentence- and token-level) to balance global context and fine-grained term distinctions. It contributes a specialized financial terminology dataset and demonstrates that TermGPT yields notable improvements over domain-pretrained and contrastive baselines on terminology QCA and QA tasks, with robust cross-domain performance. The work has practical impact for high-stakes NLP tasks in regulated domains and points to future work on adversarial robustness.

Abstract

Large language models (LLMs) have demonstrated impressive performance in text generation tasks; however, their embedding spaces often suffer from the isotropy problem, resulting in poor discrimination of domain-specific terminology, particularly in legal and financial contexts. This weakness in terminology-level representation can severely hinder downstream tasks such as legal judgment prediction or financial risk analysis, where subtle semantic distinctions are critical. To address this problem, we propose TermGPT, a multi-level contrastive fine-tuning framework designed for terminology adaptation. We first construct a sentence graph to capture semantic and structural relations, and generate semantically consistent yet discriminative positive and negative samples based on contextual and topological cues. We then devise a multi-level contrastive learning approach at both the sentence and token levels, enhancing global contextual understanding and fine-grained terminology discrimination. To support robust evaluation, we construct the first financial terminology dataset derived from official regulatory documents. Experiments show that TermGPT outperforms existing baselines in term discrimination tasks within the finance and legal domains.

Paper Structure

This paper contains 22 sections, 4 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Methodology
  5. Data Augmentation
  6. Multi-level Contrastive Learning
  7. Optimization Pipeline
  8. Experiment
  9. Experimental Settings
  10. Overall Comparison (RQ1 & RQ2)
  11. Ablation Experiment (RQ3)
  12. Performance Comparison in Different Domains (RQ4)
  13. Conclusions and Future Work
  14. Acknowledgements
  15. Dataset Collection
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Motivation Example: Terminology Misunderstanding in a Loan Application.
  • Figure 2: Overall framework of TermGPT. We first construct a sentence graph with sentences as nodes and different semantic and structural relationships as edges, where edges representing semantic ambiguity are black and lexical ambiguity edges are blue. Each node is used as an anchor sample, and its candidate samples are used for data augmentation to generate QCA pairs. Finally, contrastive learning is applied at different levels to distinguish the differences in terminology embeddings based on the QCA categories.
  • Figure 3: Comparison of different models on various datasets in terms of LLM Score.
  • Figure 4: Performance of different domains on QCA and QA tasks.