Piccolo2: General Text Embedding with Multi-task Hybrid Loss Training

TL;DR

Piccolo2 addresses the need for a versatile, high-quality Chinese text embedding by jointly training across multiple tasks with a hybrid loss that blends retrieval/reranking, STS/pair classification, and classification/clustering objectives. It boosts capacity through dimension scaling to $1792$ and enables flexible vector lengths via Matryoshka Representation Learning (MRL), augmented by a synthetic data pipeline and hard negative mining. On CMTEB, Piccolo2 achieves state-of-the-art performance, outperforming prior BERT-based embeddings by approximately $1.9$ points across six task categories. This approach offers a scalable, storage-efficient framework for robust Chinese text embeddings with practical implications for retrieval, clustering, and semantic similarity tasks.

Abstract

In this report, we introduce Piccolo2, an embedding model that surpasses other models in the comprehensive evaluation over 6 tasks on CMTEB benchmark, setting a new state-of-the-art. Piccolo2 primarily leverages an efficient multi-task hybrid loss training approach, effectively harnessing textual data and labels from diverse downstream tasks. In addition, Piccolo2 scales up the embedding dimension and uses MRL training to support more flexible vector dimensions. The latest information of piccolo models can be accessed via: https://huggingface.co/sensenova/