Out-of-the-Box Conditional Text Embeddings from Large Language Models

TL;DR

PonTE introduces a prompt-based, unsupervised method for generating conditional text embeddings using causal large language models. By embedding the target text in a one word expression conditioned on an aspect within a conditional prompt, PonTE extracts the last-token hidden state as the embedding, enabling condition-driven compression without fine tuning. Across C-STS and text clustering tasks, PonTE achieves competitive or superior performance to supervised methods and strong unsupervised baselines, while offering interpretability through generated words and embedding visualizations. This approach provides a practical baseline for unsupervised conditional embeddings with broad domain and language applicability, reducing data and computational costs associated with fine tuning.

Abstract

Conditional text embedding is a proposed representation that captures the shift in perspective on texts when conditioned on a specific aspect. Previous methods have relied on extensive training data for fine-tuning models, leading to challenges in terms of labor and resource costs. We propose PonTE, a novel unsupervised conditional text embedding method that leverages a causal large language model and a conditional prompt. Through experiments on conditional semantic text similarity and text clustering, we demonstrate that PonTE can generate useful conditional text embeddings and achieve performance comparable to supervised methods without fine-tuning. We also show the interpretability of text embeddings with PonTE by analyzing word generation following prompts and embedding visualization.

Figures (3)

• Figure 1: Visualization examples of conditional text embedding with PonTE. $T_1$, $T_2$, and $T_3$ correspond to the texts in Table \ref{['tab:examples']}.
• Figure 2: Overview of PonTE
• Figure 3: 2D t-SNE projection of embedding by PonTE