Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embedding Retrofitting: Data Engineering for better RAG

Anantha Sharma

TL;DR

This paper tackles the problem that embedding retrofitting's effectiveness hinges on the quality of the underlying knowledge graph, which is highly sensitive to text preprocessing artifacts. It introduces a data engineering pipeline (artifact removal, text normalization, filtered graph construction, and quality validation) to reduce spurious co-occurrences, demonstrating that on noisy corpora, unclean graphs derail retrofitting, while properly preprocessed graphs enable significant gains, especially for EWMA retrofitting. Through experiments on HR-1 and ZeroG, the authors show that after preprocessing, EWMA retrofitting yields statistically significant improvements (up to +6.2% with notable gains in quantitative questions), while other methods may underperform due to high variance from noise. The study further contrasts retrofitting with fine-tuning, arguing that data shaping provides a more transparent, low-cost, and deployment-friendly path to improved retrieval, and it establishes practical preprocessing quality thresholds to diagnose failures before deployment. Overall, the work highlights that data quality can dominate algorithmic sophistication in retrofitting, offering actionable guidance for enterprise RAG systems and outlining future directions for broader artifact handling and automated quality assessment.

Abstract

Embedding retrofitting adjusts pre-trained word vectors using knowledge graph constraints to improve domain-specific retrieval. However, the effectiveness of retrofitting depends critically on knowledge graph quality, which in turn depends on text preprocessing. This paper presents a data engineering framework that addresses data quality degradation from annotation artifacts in real-world corpora. The analysis shows that hashtag annotations inflate knowledge graph density, leading to creating spurious edges that corrupt the retrofitting objective. On noisy graphs, all retrofitting techniques produce statistically significant degradation ($-3.5\%$ to $-5.2\%$, $p<0.05$). After preprocessing, \acrshort{ewma} retrofitting achieves $+6.2\%$ improvement ($p=0.0348$) with benefits concentrated in quantitative synthesis questions ($+33.8\%$ average). The gap between clean and noisy preprocessing (10\%+ swing) exceeds the gap between algorithms (3\%), establishing preprocessing quality as the primary determinant of retrofitting success.

Embedding Retrofitting: Data Engineering for better RAG

TL;DR

This paper tackles the problem that embedding retrofitting's effectiveness hinges on the quality of the underlying knowledge graph, which is highly sensitive to text preprocessing artifacts. It introduces a data engineering pipeline (artifact removal, text normalization, filtered graph construction, and quality validation) to reduce spurious co-occurrences, demonstrating that on noisy corpora, unclean graphs derail retrofitting, while properly preprocessed graphs enable significant gains, especially for EWMA retrofitting. Through experiments on HR-1 and ZeroG, the authors show that after preprocessing, EWMA retrofitting yields statistically significant improvements (up to +6.2% with notable gains in quantitative questions), while other methods may underperform due to high variance from noise. The study further contrasts retrofitting with fine-tuning, arguing that data shaping provides a more transparent, low-cost, and deployment-friendly path to improved retrieval, and it establishes practical preprocessing quality thresholds to diagnose failures before deployment. Overall, the work highlights that data quality can dominate algorithmic sophistication in retrofitting, offering actionable guidance for enterprise RAG systems and outlining future directions for broader artifact handling and automated quality assessment.

Abstract

Embedding retrofitting adjusts pre-trained word vectors using knowledge graph constraints to improve domain-specific retrieval. However, the effectiveness of retrofitting depends critically on knowledge graph quality, which in turn depends on text preprocessing. This paper presents a data engineering framework that addresses data quality degradation from annotation artifacts in real-world corpora. The analysis shows that hashtag annotations inflate knowledge graph density, leading to creating spurious edges that corrupt the retrofitting objective. On noisy graphs, all retrofitting techniques produce statistically significant degradation ( to , ). After preprocessing, \acrshort{ewma} retrofitting achieves improvement () with benefits concentrated in quantitative synthesis questions ( average). The gap between clean and noisy preprocessing (10\%+ swing) exceeds the gap between algorithms (3\%), establishing preprocessing quality as the primary determinant of retrofitting success.
Paper Structure (22 sections, 1 theorem, 7 equations, 12 figures, 7 tables)

This paper contains 22 sections, 1 theorem, 7 equations, 12 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Scope
  4. Datasets
  5. Related Work
  6. The Data Engineering Pipeline
  7. Retrofitting Techniques
  8. Temporal Regularization vs. Instantaneous Reweighting
  9. Experimental Setup
  10. Quantifying Preprocessing Effects
  11. Recovering Retrofitting Benefits
  12. Ablating Pipeline Components
  13. Question-Type Analysis
  14. Response Examples
  15. Embedding Space Visualization
  16. ...and 7 more sections

Key Result

Proposition 5.1

Let $\sigma^2$ denote the variance of single-iteration updates. Under EWMA with decay $\beta$, the variance of the smoothed trajectory is: For $\beta = 0.8$, this yields a variance reduction factor of $\approx 0.11$, explaining the observed coefficient of variation improvement (1.9% vs. 3.0% for Regular).

Figures (12)

  • Figure 1: Update magnitude trajectories across retrofitting iterations. Attention exhibits high-frequency oscillations due to instantaneous weight recomputation. EWMA's temporal smoothing produces monotonic convergence with lower variance, enabling statistical significance with smaller sample sizes.
  • Figure 2: Retrofitting quality change ($\Delta$Q) on ZeroG corpus. Raw data (red) shows degradation across all techniques; after data engineering (green) all techniques improve. EWMA achieves statistical significance ($p=0.041$).
  • Figure 3: Cumulative preprocessing impact on EWMA retrofitting (ZeroG corpus). Each stage progressively improves results. Red indicates statistically significant degradation; green indicates significant improvement. * denotes $p < 0.05$.
  • Figure 4: EWMA retrofitting improvement by question type (legislative corpus). Bars show mean improvement; triangles indicate maximum improvement. Quantitative questions requiring numerical synthesis benefit most from retrofitted embeddings.
  • Figure 5: Response comparison across question types. Highlighted tokens indicate quality-improving content: specific numbers, source attributions, and domain terminology retrieved through retrofitted embeddings.
  • ...and 7 more figures

Theorems & Definitions (1)

  • Proposition 5.1: EWMA Variance Reduction