Harmonizing Diverse Models: A Layer-wise Merging Strategy for Consistent Generation

TL;DR

This work tackles the problem of output inconsistency in retrieval-augmented generation by introducing a triad of methods: systematic synthetic data generation to cover diverse input variations, triplet loss training to align semantically similar queries in embedding space, and a novel layer-wise model merging strategy guided by consistency-oriented weights derived from intermediate activations. The approach builds multiple specialized models trained on different synthetic data types and merges them using a DARE-TIES-inspired mechanism that weights each model’s contribution per layer based on activation similarity. Empirical results on industrial RAG tasks show substantial gains in consistency metrics (e.g., a ~47.5% improvement in Response Similarity over the baseline) while maintaining strong accuracy, illustrating a practical pathway to more reliable LLMs in production settings. The work also provides a framework for adaptive, per-layer merging and highlights directions for public benchmarks and broader applicability to retriever-affected inconsistencies.

Abstract

Retrieval-Augmented Generation (RAG) systems leverage Large Language Models (LLMs) to generate accurate and reliable responses that are grounded in retrieved context. However, LLMs often generate inconsistent outputs for semantically equivalent inputs, a problem compounded by the scarcity of consistency-focused training data and the limitations of current fine-tuning techniques in enhancing output consistency. We propose a new approach combining systematic synthetic data generation, triplet loss for better embeddings, and a novel layer-wise model merging approach. Using consistency-aware weights derived from intermediate layer activations, our method effectively integrates knowledge from specialized models. Experimental results how that our merged model significantly enhances output consistency, achieving a ~47.5\% improvement in response similarity over the baseline, thus offering a practical solution for increasing the reliability of an industrial RAG system.

Figures (1)

• Figure 1: Variability in LLM Responses from Subtle Query Differences.