Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hakim: Farsi Text Embedding Model

Mehran Sarmadi, Morteza Alikhani, Erfan Zinvandi, Zahra Pourbahman

TL;DR

This work addresses the underrepresentation of Persian in large-scale text embeddings by introducing Hakim, a state-of-the-art Persian embedding model. It combines the Corpesia corpus with large-scale paired data (Pairsia-unsup and Pairsia-sup) and utilizes a RetroMAE-inspired, two-stage training regime that includes instruction-based supervision. On FaMTEB, Hakim achieves an $8.5\%$ improvement over prior Persian models and demonstrates strong capabilities in chatbots, retrieval-augmented generation, zero-shot tasks, and cross-classification. The contributions include the Corpesia and Pairsia datasets, a BERT-based embedding model tuned for retrieval and downstream tasks, and evidence that instruction-tuning and RAG-oriented training substantially boost practical applicability in Persian NLP.

Abstract

Recent advancements in text embedding have significantly improved natural language understanding across many languages, yet Persian remains notably underrepresented in large-scale embedding research. In this paper, we present Hakim, a novel state-of-the-art Persian text embedding model that achieves a 8.5% performance improvement over existing approaches on the FaMTEB benchmark, outperforming all previously developed Persian language models. As part of this work, we introduce three new datasets - Corpesia, Pairsia-sup, and Pairsia-unsup - to support supervised and unsupervised training scenarios. Additionally, Hakim is designed for applications in chatbots and retrieval-augmented generation (RAG) systems, particularly addressing retrieval tasks that require incorporating message history within these systems. We also propose a new baseline model built on the BERT architecture. Our language model consistently achieves higher accuracy across various Persian NLP tasks, while the RetroMAE-based model proves particularly effective for textual information retrieval applications. Together, these contributions establish a new foundation for advancing Persian language understanding.

Hakim: Farsi Text Embedding Model

TL;DR

This work addresses the underrepresentation of Persian in large-scale text embeddings by introducing Hakim, a state-of-the-art Persian embedding model. It combines the Corpesia corpus with large-scale paired data (Pairsia-unsup and Pairsia-sup) and utilizes a RetroMAE-inspired, two-stage training regime that includes instruction-based supervision. On FaMTEB, Hakim achieves an improvement over prior Persian models and demonstrates strong capabilities in chatbots, retrieval-augmented generation, zero-shot tasks, and cross-classification. The contributions include the Corpesia and Pairsia datasets, a BERT-based embedding model tuned for retrieval and downstream tasks, and evidence that instruction-tuning and RAG-oriented training substantially boost practical applicability in Persian NLP.

Abstract

Recent advancements in text embedding have significantly improved natural language understanding across many languages, yet Persian remains notably underrepresented in large-scale embedding research. In this paper, we present Hakim, a novel state-of-the-art Persian text embedding model that achieves a 8.5% performance improvement over existing approaches on the FaMTEB benchmark, outperforming all previously developed Persian language models. As part of this work, we introduce three new datasets - Corpesia, Pairsia-sup, and Pairsia-unsup - to support supervised and unsupervised training scenarios. Additionally, Hakim is designed for applications in chatbots and retrieval-augmented generation (RAG) systems, particularly addressing retrieval tasks that require incorporating message history within these systems. We also propose a new baseline model built on the BERT architecture. Our language model consistently achieves higher accuracy across various Persian NLP tasks, while the RetroMAE-based model proves particularly effective for textual information retrieval applications. Together, these contributions establish a new foundation for advancing Persian language understanding.
Paper Structure (24 sections, 6 equations, 6 figures, 11 tables)

This paper contains 24 sections, 6 equations, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General Text Embedding Approaches
  4. Persian Text Embedding
  5. FaMTEB
  6. Method
  7. Data
  8. Pre-training Data
  9. Pairsia-unsup
  10. Pairsia-sup
  11. Model
  12. Pretraining
  13. Stage 1 (Unsupervised)
  14. Stage 2 (Supervised)
  15. Training Details
  16. ...and 9 more sections

Figures (6)

  • Figure 1: This figure presents an overview of all the datasets utilized for training the Hakim network. The Corpesia corpus is employed for training the base model. The Pairsia-unsup dataset is used during the unsupervised training phase, while the Pairsia-sup dataset is incorporated in the final stage of training.
  • Figure 2: The template of adding instructions to different data types used in Hakim. In general, data is categorized into three types: classification, cross-classification, and Relative Pairs types. Instructions and data pairs are added accordingly.
  • Figure 3: The instructions employed for addressing classification tasks.
  • Figure 4: The instructions employed for addressing cross classification tasks.
  • Figure 5: The instructions employed for addressing Relative Pair tasks like retrieval, sts, and summarization.
  • ...and 1 more figures