Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GigaEmbeddings: Efficient Russian Language Embedding Model

Egor Kolodin, Daria Khomich, Nikita Savushkin, Anastasia Ianina, Fyodor Minkin

TL;DR

GigaEmbeddings tackles the challenge of robust Russian language embeddings by leveraging a decoder-only LLM (GigaChat-3B) in a three-stage instruction-tuning framework that combines large-scale contrastive pretraining, hard-negative fine-tuning, and multitask generalization across retrieval, classification, and clustering. The model introduces architectural innovations—bidirectional attention, latent attention pooling, and 25% layer pruning—to achieve efficiency without sacrificing performance. Evaluation on ruMTEB across 23 tasks yields a state-of-the-art average of 69.1, surpassing strong baselines and confirming the viability of decoder-only LLMs as universal encoders for multilingual, multitask embedding tasks. The work demonstrates the power of synthetic data generation and task-aware instruction tuning for high-quality, scalable embeddings in Russian settings, with open-source release enabling broader adoption and future extensions to low-resource languages.

Abstract

We introduce GigaEmbeddings, a novel framework for training high-performance Russian-focused text embeddings through hierarchical instruction tuning of the decoder-only LLM designed specifically for Russian language (GigaChat-3B). Our three-stage pipeline, comprising large-scale contrastive pre-training in web-scale corpora, fine-tuning with hard negatives, and multitask generalization across retrieval, classification, and clustering tasks, addresses key limitations of existing methods by unifying diverse objectives and leveraging synthetic data generation. Architectural innovations include bidirectional attention for contextual modeling, latent attention pooling for robust sequence aggregation, and strategic pruning of 25% of transformer layers to enhance efficiency without compromising performance. Evaluated on the ruMTEB benchmark spanning 23 multilingual tasks, GigaEmbeddings achieves state-of-the-art results (69.1 avg. score), outperforming strong baselines with a larger number of parameters.

GigaEmbeddings: Efficient Russian Language Embedding Model

TL;DR

GigaEmbeddings tackles the challenge of robust Russian language embeddings by leveraging a decoder-only LLM (GigaChat-3B) in a three-stage instruction-tuning framework that combines large-scale contrastive pretraining, hard-negative fine-tuning, and multitask generalization across retrieval, classification, and clustering. The model introduces architectural innovations—bidirectional attention, latent attention pooling, and 25% layer pruning—to achieve efficiency without sacrificing performance. Evaluation on ruMTEB across 23 tasks yields a state-of-the-art average of 69.1, surpassing strong baselines and confirming the viability of decoder-only LLMs as universal encoders for multilingual, multitask embedding tasks. The work demonstrates the power of synthetic data generation and task-aware instruction tuning for high-quality, scalable embeddings in Russian settings, with open-source release enabling broader adoption and future extensions to low-resource languages.

Abstract

We introduce GigaEmbeddings, a novel framework for training high-performance Russian-focused text embeddings through hierarchical instruction tuning of the decoder-only LLM designed specifically for Russian language (GigaChat-3B). Our three-stage pipeline, comprising large-scale contrastive pre-training in web-scale corpora, fine-tuning with hard negatives, and multitask generalization across retrieval, classification, and clustering tasks, addresses key limitations of existing methods by unifying diverse objectives and leveraging synthetic data generation. Architectural innovations include bidirectional attention for contextual modeling, latent attention pooling for robust sequence aggregation, and strategic pruning of 25% of transformer layers to enhance efficiency without compromising performance. Evaluated on the ruMTEB benchmark spanning 23 multilingual tasks, GigaEmbeddings achieves state-of-the-art results (69.1 avg. score), outperforming strong baselines with a larger number of parameters.
Paper Structure (19 sections, 2 equations, 1 figure, 5 tables)

This paper contains 19 sections, 2 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Leveraging Pretrained Language Models
  4. Efficient Architectures and Long-Context Modeling
  5. Synthetic Data for Training Embedding Models
  6. Method
  7. Training Objective
  8. Three-Stage Instruction-Tuning
  9. Pretraining with Large-Batch Contrastive Learning
  10. Fine-Tuning with Hard Negatives
  11. Multitask Generalization
  12. Architectural Innovations
  13. Experiments
  14. Baselines
  15. Main results
  16. ...and 4 more sections

Figures (1)

  • Figure 1: The design comprises a GigaChat LLM followed by a latent attention layer. This layer operates through a cross-attention mechanism where the decoder’s output acts as the query (Q), a trainable latent array provides the key (K) and value (V) inputs. The attention output is then processed by a multilayer perceptron (MLP).