DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management

TL;DR

DMRetriever addresses disaster management information needs by delivering a domain-specific dense retrieval family (33M–7.6B) trained with a novel data refinement pipeline and a three-stage learning framework. It combines bidirectional attention adaptation for decoder-only backbones, unsupervised contrastive pre-training on DM-MTP/GD-MTP, and difficulty-aware progressive instruction fine-tuning to cover diverse intents. Across six disaster-related search intents, it achieves state-of-the-art results while maintaining high parameter efficiency, with the 596M model outperforming XL baselines and the 33M model surpassing medium baselines. The approach demonstrates strong data efficiency and scalability, and the authors provide open-source data, models, and code to support reproducibility and broader domain adaptation.

Abstract

Effective and efficient access to relevant information is essential for disaster management. However, no retrieval model is specialized for disaster management, and existing general-domain models fail to handle the varied search intents inherent to disaster management scenarios, resulting in inconsistent and unreliable performance. To this end, we introduce DMRetriever, the first series of dense retrieval models (33M to 7.6B) tailored for this domain. It is trained through a novel three-stage framework of bidirectional attention adaptation, unsupervised contrastive pre-training, and difficulty-aware progressive instruction fine-tuning, using high-quality data generated through an advanced data refinement pipeline. Comprehensive experiments demonstrate that DMRetriever achieves state-of-the-art (SOTA) performance across all six search intents at every model scale. Moreover, DMRetriever is highly parameter-efficient, with 596M model outperforming baselines over 13.3 X larger and 33M model exceeding baselines with only 7.6% of their parameters. All codes, data, and checkpoints are available at https://github.com/KaiYin97/DMRETRIEVER

Figures (11)

• Figure 1: Average performance of DMRetriever. Breakdown for each search intent is in Table \ref{['tab:embedding_results']}
• Figure 2: Overview of data refinement pipeline and progressive fine-tuning for DMRetriever. MTP: Massive Text Pairs, MTT: Massive Text Triplets. Both MTP and MTT in sub-figure (B) include DM (Disaster Management) and GD (General Domain) variants, with constructions of GD variants omitted here and detailed in §\ref{['section: pt_dataset_construction']} and §\ref{['sec: hard_neg_mine']}. DM passages include those extracted from PDFs and those generated by LLMs. DMRetriever-Init: backbone after bidirectional attention adaptation for decoder-only; original backbone for encoder-only. DMRetriever-PT: model after pre-training. DMRetriever-It2: model after iteration 2. Algorithm \ref{['alg:dmretriever_train']} details the full workflow.
• Figure 3: Effects of different fine-tuning datasets. DM-MTT and GD-MTT are datasets used during fine-tuning from disaster management and general domains. Effects of different pre-training datasets are shown in Figure \ref{['fig:PT_ablation']}.
• Figure 4: Ablation study of different false positive filtering methods. NoPos: No false positive filtering. CBF: Consistency-based filtering. T1--3: Top-$N$ mutual-agreement filtering strategy, where $N=1, 2, 3$.
• Figure 5: Ablations of progressive instruction fine-tuning. DirectIt3, BiAttn-Ada, DM-PT, and Ins mean fine-tuning directly using iteration 3 data (MTT-0.85), Bidirectional attention adaptation, DMRetriever model, and instruction, respectively. Breakdown for each search intent is in Figures \ref{['fig:Ablation_Stage_eachtask_FT']}, \ref{['fig:Ablation_Stage_eachtask_PTS1']}, and \ref{['fig:Ablation_Stage_eachtask_PT_vs_noPT']}.