Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weighting

Polina Proskura, Alexey Zaytsev

TL;DR

The paper tackles improving NLP ensembles by accounting for inter-model diversity through topology-based similarity measures computed from attention representations. It formalizes a quadratic-risk weighting framework and uses RTD-based topology divergences to determine ensemble weights, achieving higher accuracy and tighter uncertainty estimates than equal-weight or simple correlation-based methods. Across IMDb, CoLA, SST-2, and MRPC, topology-aware weighting yields consistent gains and enables effective subset selection, including scenarios with weak and strong model pairings. The approach offers practical benefits by boosting performance without training additional models and shows potential for broader applications, such as joint training and translation tasks.

Abstract

In machine learning, ensembles are important tools for improving the model performance. In natural language processing specifically, ensembles boost the performance of a method due to multiple large models available in open source. However, existing approaches mostly rely on simple averaging of predictions by ensembles with equal weights for each model, ignoring differences in the quality and conformity of models. We propose to estimate weights for ensembles of NLP models using not only knowledge of their individual performance but also their similarity to each other. By adopting distance measures based on Topological Data Analysis (TDA), we improve our ensemble. The quality improves for both text classification accuracy and relevant uncertainty estimation.

Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weighting

TL;DR

The paper tackles improving NLP ensembles by accounting for inter-model diversity through topology-based similarity measures computed from attention representations. It formalizes a quadratic-risk weighting framework and uses RTD-based topology divergences to determine ensemble weights, achieving higher accuracy and tighter uncertainty estimates than equal-weight or simple correlation-based methods. Across IMDb, CoLA, SST-2, and MRPC, topology-aware weighting yields consistent gains and enables effective subset selection, including scenarios with weak and strong model pairings. The approach offers practical benefits by boosting performance without training additional models and shows potential for broader applications, such as joint training and translation tasks.

Abstract

In machine learning, ensembles are important tools for improving the model performance. In natural language processing specifically, ensembles boost the performance of a method due to multiple large models available in open source. However, existing approaches mostly rely on simple averaging of predictions by ensembles with equal weights for each model, ignoring differences in the quality and conformity of models. We propose to estimate weights for ensembles of NLP models using not only knowledge of their individual performance but also their similarity to each other. By adopting distance measures based on Topological Data Analysis (TDA), we improve our ensemble. The quality improves for both text classification accuracy and relevant uncertainty estimation.
Paper Structure (32 sections, 3 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 32 sections, 3 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Ensembles
  4. Ensembles in NLP
  5. Attention layer features
  6. Uncertainty estimation
  7. Research gap
  8. Methods
  9. Weighting
  10. Optimization Algorithms
  11. Similarity measure for models
  12. Models' output correlation
  13. Topology Features-Based Divergence
  14. Barcodes
  15. R-Cross-Barcode
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Example of Barcode for Four Features With Different Time $\tau$ of Deletion.
  • Figure 2: Example of Two Attention Graphs With Matched Vertices and Their Union Graph.
  • Figure 3: Rejection Curves for IMDB (Left) and CoLA (Right) Datasets for DistilBERT model. The Blue curves correspond to Individual Models, the Red curve Is for Equal Weights Ensemble, the Orange Is for the Output-Based-Correlation Model and the Green Curve Is for the Ensemble Optimized Using the RTD-Based Correlation.