Learning High-Quality and General-Purpose Phrase Representations

TL;DR

PEARL tackles the challenge of learning high-quality phrase representations from context-free input by integrating phrase-type information and robust character-level features into a contrastive learning framework. It combines a phrase encoder based on E5 with a LOVE-inspired character encoder, augments data at character, token, and phrase levels, and uses an auxiliary Phrase Type Classification task to sharpen semantic distinctions, all trained with hard negatives and weight averaging. The approach yields strong improvements across diverse phrase and short-text tasks, achieving state-of-the-art performance with smaller models (e.g., PEARL-small at 40M parameters) and generalizes well to multiple backbone language models. This work offers practical benefits for entity retrieval, clustering, and fuzzy matching in settings with limited contextual cues, while highlighting the importance of morphology and type information in phrase representations.

Abstract

Phrase representations play an important role in data science and natural language processing, benefiting various tasks like Entity Alignment, Record Linkage, Fuzzy Joins, and Paraphrase Classification. The current state-of-the-art method involves fine-tuning pre-trained language models for phrasal embeddings using contrastive learning. However, we have identified areas for improvement. First, these pre-trained models tend to be unnecessarily complex and require to be pre-trained on a corpus with context sentences. Second, leveraging the phrase type and morphology gives phrase representations that are both more precise and more flexible. We propose an improved framework to learn phrase representations in a context-free fashion. The framework employs phrase type classification as an auxiliary task and incorporates character-level information more effectively into the phrase representation. Furthermore, we design three granularities of data augmentation to increase the diversity of training samples. Our experiments across a wide range of tasks show that our approach generates superior phrase embeddings compared to previous methods while requiring a smaller model size. [PEARL-small]: https://huggingface.co/Lihuchen/pearl_small; [PEARL-base]: https://huggingface.co/Lihuchen/pearl_base; [Code and Dataset]: https://github.com/tigerchen52/PEARL

Figures (6)

• Figure 1: An example of entity retrieval. Given the input entity name "The New York Times", we show the cosine similarity obtained by different models. The ranking of scores is listed in parentheses.
• Figure 2: An illustration of PEARL. It uses contrastive learning and an auxiliary task of phrase type prediction for learning phrase embeddings.
• Figure 3: Different levels of granularity for the data augmentation methods on "The New York Times".
• Figure 4: t-SNE visualizations of phrase embeddings generated by different models. We randomly selected 100 samples for each entity type from YAGO 4 (Place, Person, MeidicalEntity, Event, Organization, CreativeWork). Markers with the same color are supposed to be grouped together.
• Figure 5: Hyperparameter selection on BIRD dataset.