TNCSE: Tensor's Norm Constraints for Unsupervised Contrastive Learning of Sentence Embeddings

TL;DR

This work tackles the limitation of cosine-only metrics in unsupervised sentence embeddings by introducing Tensor Norm Constraints (TNCSE) that jointly optimize alignment and magnitude between positive samples. The method uses a dual-encoder ensemble with three losses—InfoNCE ($L_{NCE}$), interaction-constrained InfoNCE ($L_{ICNCE}$), and interaction-constrained Tensor Norm ($L_{ICTN}$)—and a norm-based objective $l_{TN}$ to constrain embedding magnitudes, with $l_{TN}(k,t) = \frac{\sqrt{1+k^{2}-2kt}}{1+k}$ when $||h^{+}|| = k||h||$ and $t=\cos \gamma$. Evaluated on seven STS tasks and extensive zero-shot MTEB benchmarks, TNCSE achieves state-of-the-art results and demonstrates robustness via ablations and significance tests, while offering improved efficiency over larger ensembles. The findings show that combining norm-based constraints with ensemble learning yields more discriminative sentence representations with practical inference costs, benefiting downstream tasks and multilingual settings.

Abstract

Unsupervised sentence embedding representation has become a hot research topic in natural language processing. As a tensor, sentence embedding has two critical properties: direction and norm. Existing works have been limited to constraining only the orientation of the samples' representations while ignoring the features of their module lengths. To address this issue, we propose a new training objective that optimizes the training of unsupervised contrastive learning by constraining the module length features between positive samples. We combine the training objective of Tensor's Norm Constraints with ensemble learning to propose a new Sentence Embedding representation framework, TNCSE. We evaluate seven semantic text similarity tasks, and the results show that TNCSE and derived models are the current state-of-the-art approach; in addition, we conduct extensive zero-shot evaluations, and the results show that TNCSE outperforms other baselines.

Figures (6)

• Figure 1: The figure denotes the subtraction of the semantic representation tensor in 3D space.
• Figure 2: The figure denotes the binary function image of the constraint loss of the tensor norm for independent variables $k$ and $t$.
• Figure 3: The left side of the solid line is the traditional ensemble learning method represented by EDFSE, and the right is our proposed new method based on semantic tensor with norm constraints. When TNCSE is training, sample $A$ passes through two encoders simultaneously, obtains the last hidden state, does InfoNCE$(L_{NCE})$ and ICNCE$(L_{ICNCE})$, then passes through the corresponding pooler layer and does TNC$(L_{ICTN})$ with norm constraints crossly, respectively. When TNCSE is inference, sample $A$ passes through two encoders simultaneously, and then the two last hidden states are directly summed up as the output.
• Figure 4: The figure reports the results of the significance test. We specify the random seeds are 1 to 5, other hyperparameters are defaulted, training set is uniformly Wiki1M and unlabelled SICKR.
• Figure 5: The bar and line graphs separately represent the norm mean of the Last hidden state and the model's performance on the STS-B validation set for different LayerNorm settings.