Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing

TL;DR

This work tackles the challenge of domain-adapting large language models to mortgage finance without sacrificing instruction-following. It introduces MortgageLLM, a dual-track specialization built on a base LLaMA-3.1-8B model, using Continued Pretraining, Instruction Residual, and Direct Preference Optimization, plus an intelligent self-routing mechanism to allocate queries between a conversational Q&A specialist and a structured-task specialist. MLM v2 demonstrates superior domain alignment and task performance across summarization, classification, and Q&A, surpassing baselines in semantic fidelity, judge-based evaluations, and SME preferences, while improving security metrics. The approach offers a scalable, efficient pathway to production-ready, domain-specific LLMs for regulated financial domains, with clear avenues for expansion to multi-modal data and more granular MoE routing.

Abstract

Large Language Models (LLMs) demonstrate exceptional capabilities across general domains, yet their application to specialized sectors such as mortgage finance requires domain-specific knowledge augmentation while preserving instruction-following fidelity. We present MortgageLLM, a novel domain-specific large language model that addresses this dual challenge. It is developed using a dual-track specialization framework from a single base model (LLaMA-3.1-8B). We opted for this dual-expert approach as a single multi-task model suffers from performance trade-offs, where optimizing for structured tasks (via SFT) degrades conversational fidelity (via DPO). Our dual-track method solves this by creating two specialists, allowing each to be optimally trained for its distinct capability. Our approach applies the instruction residual technique to restore instruction-following capabilities post-domain adaptation without supervised fine-tuning. We contribute: (1) application of this residual technique to the highly specialized mortgage finance domain; (2) a dual-expert architecture combining a conversational Q&A model and a structured task model for classification and summarization; and (3) an intelligent task routing mechanism using few-shot classification performed by one of the expert models itself. We validate our approach on domain-specific benchmarks, where our final model (MLM v2) significantly outperforms the base LLaMA-3.1-8B-Instruct, achieving an LLM-as-a-Judge summarization score of 4.58 (vs. 3.99), a Q&A score of 4.09 (vs. 4.0), and a classification score of 2.6 (vs. 1.2). On semantic similarity, our model achieved a BERTScore of 0.77 for summarization (vs. 0.74), 0.68 for Q&A (vs. 0.58), and 0.75 for classification (vs. 0.73), substantially outperforming baseline approaches.

Figures (7)

• Figure 1: Architectural flow for Track 1 (Residual Learning).
• Figure 2: Architectural flow for Track 2 (Direct Fine-tuning).
• Figure 3: BERTScore Evaluation Results (Higher is better)
• Figure 4: LLM-as-a-Judge Evaluation Results (Higher is better)
• Figure 5: SME Model Evaluation Distribution