Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Sentence Embedding with Pseudo-Labeling and Model Ensembles: A Hierarchical Framework for Enhanced NLP Tasks

Ziwei Liu, Qi Zhang, Lifu Gao

TL;DR

This paper addresses the challenge of producing reliable sentence embeddings when data are limited and domain variation is high. It introduces a hierarchical framework that fuses pseudo-labeling, external corpora (SimpleWiki, Wikipedia, BookCorpus), and model ensembles using backbones ALBERT-xxlarge, RoBERTa-large, and DeBERTa-large. Key contributions include a three-stage model (encoding, refinement, ensemble), cross-attention with external context, data augmentation, and an error-aware pseudo-labeling pipeline, yielding substantial gains over baselines. The approach provides a robust, generalizable blueprint for enhanced sentence embeddings applicable to cross-lingual and real-time NLP tasks.

Abstract

Sentence embedding tasks are important in natural language processing (NLP), but improving their performance while keeping them reliable is still hard. This paper presents a framework that combines pseudo-label generation and model ensemble techniques to improve sentence embeddings. We use external data from SimpleWiki, Wikipedia, and BookCorpus to make sure the training data is consistent. The framework includes a hierarchical model with an encoding layer, refinement layer, and ensemble prediction layer, using ALBERT-xxlarge, RoBERTa-large, and DeBERTa-large models. Cross-attention layers combine external context, and data augmentation techniques like synonym replacement and back-translation increase data variety. Experimental results show large improvements in accuracy and F1-score compared to basic models, and studies confirm that cross-attention and data augmentation make a difference. This work presents an effective way to improve sentence embedding tasks and lays the groundwork for future NLP research.

Optimizing Sentence Embedding with Pseudo-Labeling and Model Ensembles: A Hierarchical Framework for Enhanced NLP Tasks

TL;DR

This paper addresses the challenge of producing reliable sentence embeddings when data are limited and domain variation is high. It introduces a hierarchical framework that fuses pseudo-labeling, external corpora (SimpleWiki, Wikipedia, BookCorpus), and model ensembles using backbones ALBERT-xxlarge, RoBERTa-large, and DeBERTa-large. Key contributions include a three-stage model (encoding, refinement, ensemble), cross-attention with external context, data augmentation, and an error-aware pseudo-labeling pipeline, yielding substantial gains over baselines. The approach provides a robust, generalizable blueprint for enhanced sentence embeddings applicable to cross-lingual and real-time NLP tasks.

Abstract

Sentence embedding tasks are important in natural language processing (NLP), but improving their performance while keeping them reliable is still hard. This paper presents a framework that combines pseudo-label generation and model ensemble techniques to improve sentence embeddings. We use external data from SimpleWiki, Wikipedia, and BookCorpus to make sure the training data is consistent. The framework includes a hierarchical model with an encoding layer, refinement layer, and ensemble prediction layer, using ALBERT-xxlarge, RoBERTa-large, and DeBERTa-large models. Cross-attention layers combine external context, and data augmentation techniques like synonym replacement and back-translation increase data variety. Experimental results show large improvements in accuracy and F1-score compared to basic models, and studies confirm that cross-attention and data augmentation make a difference. This work presents an effective way to improve sentence embedding tasks and lays the groundwork for future NLP research.
Paper Structure (23 sections, 17 equations, 4 figures, 2 tables)

This paper contains 23 sections, 17 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Model Network
  5. Encoding Layer
  6. Refinement Layer
  7. Ensemble Prediction Layer
  8. Loss Function
  9. Data Preprocessing
  10. Data Collection
  11. Snippet Generation
  12. Embedding-Based Retrieval
  13. Pseudo-Labeling and Error-Based Filtering
  14. Augmentation Strategies
  15. Final Data Preparation
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Leveraging hybrid model architectures pipeline
  • Figure 2: The pipline and detail in data preprocessing
  • Figure 3: The top 5 most similar fragments.
  • Figure 4: The changge metrics in training processing.