Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Skim-Aware Contrastive Learning for Efficient Document Representation

Waheed Ahmed Abro, Zied Bouraoui

TL;DR

This work tackles the challenge of producing high-quality long-document embeddings in domains like law and medicine, where standard transformer encoders struggle with length and context. It introduces Chunk Prediction Encoders (CPE), a self-supervised, contrastive-learning framework that samples text chunks and uses an NLI-inspired objective to align related chunks while separating unrelated ones, enabling richer document representations with improved efficiency. By applying CPE to hierarchical transformers and Longformer, and evaluating on legal and biomedical datasets, the authors demonstrate consistent gains in macro- and micro-F1 scores, particularly when using domain-specific pretrained models (e.g., LegalBERT, ClinicalBioBERT). The results suggest that modeling intra- and inter-document chunk relationships, combined with selective chunking, yields practical improvements for long-document classification and can reduce training time relative to some contrastive baselines, with potential for broader cross-domain and multilingual applicability.

Abstract

Although transformer-based models have shown strong performance in word- and sentence-level tasks, effectively representing long documents, especially in fields like law and medicine, remains difficult. Sparse attention mechanisms can handle longer inputs, but are resource-intensive and often fail to capture full-document context. Hierarchical transformer models offer better efficiency but do not clearly explain how they relate different sections of a document. In contrast, humans often skim texts, focusing on important sections to understand the overall message. Drawing from this human strategy, we introduce a new self-supervised contrastive learning framework that enhances long document representation. Our method randomly masks a section of the document and uses a natural language inference (NLI)-based contrastive objective to align it with relevant parts while distancing it from unrelated ones. This mimics how humans synthesize information, resulting in representations that are both richer and more computationally efficient. Experiments on legal and biomedical texts confirm significant gains in both accuracy and efficiency.

Skim-Aware Contrastive Learning for Efficient Document Representation

TL;DR

This work tackles the challenge of producing high-quality long-document embeddings in domains like law and medicine, where standard transformer encoders struggle with length and context. It introduces Chunk Prediction Encoders (CPE), a self-supervised, contrastive-learning framework that samples text chunks and uses an NLI-inspired objective to align related chunks while separating unrelated ones, enabling richer document representations with improved efficiency. By applying CPE to hierarchical transformers and Longformer, and evaluating on legal and biomedical datasets, the authors demonstrate consistent gains in macro- and micro-F1 scores, particularly when using domain-specific pretrained models (e.g., LegalBERT, ClinicalBioBERT). The results suggest that modeling intra- and inter-document chunk relationships, combined with selective chunking, yields practical improvements for long-document classification and can reduce training time relative to some contrastive baselines, with potential for broader cross-domain and multilingual applicability.

Abstract

Although transformer-based models have shown strong performance in word- and sentence-level tasks, effectively representing long documents, especially in fields like law and medicine, remains difficult. Sparse attention mechanisms can handle longer inputs, but are resource-intensive and often fail to capture full-document context. Hierarchical transformer models offer better efficiency but do not clearly explain how they relate different sections of a document. In contrast, humans often skim texts, focusing on important sections to understand the overall message. Drawing from this human strategy, we introduce a new self-supervised contrastive learning framework that enhances long document representation. Our method randomly masks a section of the document and uses a natural language inference (NLI)-based contrastive objective to align it with relevant parts while distancing it from unrelated ones. This mimics how humans synthesize information, resulting in representations that are both richer and more computationally efficient. Experiments on legal and biomedical texts confirm significant gains in both accuracy and efficiency.
Paper Structure (26 sections, 4 equations, 4 figures, 9 tables)

This paper contains 26 sections, 4 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Modeling of long documents.
  4. Unsupervised Document Representation.
  5. Legal and medical document representation
  6. Chunk Prediction Encoders
  7. CPE for Hierarchical Representation.
  8. Learning process
  9. CPE for Longformer
  10. Document Classification
  11. Experimental Setting
  12. Datasets
  13. Setting and evaluation metrics
  14. Document Embedding Baseline Models
  15. End-to-end Finetuning Baseline Models
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Illustration of CPE Contrastive Learning via the hierarchical transformer model.
  • Figure 2: The CPE Contrastive Learning process with the Longformer model.
  • Figure 3: Macro-F1 Scores vs. Chunk Size via hierarchical transformer encoders applied on LegalBERT for ECHR, SCOTUS, EURLEX, and ClinicalBioBERT for MIMIC datasets.
  • Figure 4: t-SNE visualization on the Scotus dataset with corresponding legal class.