Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

Yijie Hong, Xiaofei Yin, Xinzhong Wang, Yi Tu, Ya Guo, Sufeng Duan, Weiqiang Wang, Lingyong Fang, Depeng Wang, Huijia Zhu

TL;DR

This work tackles the challenge of injecting domain-specific knowledge into large vision-language models without catastrophic forgetting. It introduces Structured Dialogue Fine-Tuning (SDFT), a data-centric, three-phase dialogue framework comprising Foundation Preservation, Contrastive Disambiguation, and Knowledge Specialization, combined with a weighted multi-turn supervision scheme. Across personalized entities, abstract concepts, and biomedical domain knowledge, SDFT demonstrates strong knowledge acquisition while preserving general visual-linguistic capabilities, with extensive ablations illustrating the contribution of each component. The approach is model-agnostic and scalable, offering a practical path to domain-adapted LVLMs suitable for real-world deployment where both accuracy and broad capability retention matter.

Abstract

Large Vision Language Models have demonstrated impressive versatile capabilities through extensive multimodal pre-training, but face significant limitations when incorporating specialized knowledge domains beyond their training distribution. These models struggle with a fundamental dilemma: direct adaptation approaches that inject domain-specific knowledge often trigger catastrophic forgetting of foundational visual-linguistic abilities. We introduce Structured Dialogue Fine-Tuning (SDFT), an effective approach that effectively injects domain-specific knowledge while minimizing catastrophic forgetting. Drawing inspiration from supervised fine-tuning in LLMs and subject-driven personalization in text-to-image diffusion models, our method employs a three-phase dialogue structure: Foundation Preservation reinforces pre-trained visual-linguistic alignment through caption tasks; Contrastive Disambiguation introduces carefully designed counterfactual examples to maintain semantic boundaries; and Knowledge Specialization embeds specialized information through chain-of-thought reasoning. Experimental results across multiple domains confirm SDFT's effectiveness in balancing specialized knowledge acquisition with general capability retention. Our key contributions include a data-centric dialogue template that balances foundational alignment with targeted knowledge integration, a weighted multi-turn supervision framework, and comprehensive evaluation across diverse knowledge types.

Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

TL;DR

This work tackles the challenge of injecting domain-specific knowledge into large vision-language models without catastrophic forgetting. It introduces Structured Dialogue Fine-Tuning (SDFT), a data-centric, three-phase dialogue framework comprising Foundation Preservation, Contrastive Disambiguation, and Knowledge Specialization, combined with a weighted multi-turn supervision scheme. Across personalized entities, abstract concepts, and biomedical domain knowledge, SDFT demonstrates strong knowledge acquisition while preserving general visual-linguistic capabilities, with extensive ablations illustrating the contribution of each component. The approach is model-agnostic and scalable, offering a practical path to domain-adapted LVLMs suitable for real-world deployment where both accuracy and broad capability retention matter.

Abstract

Large Vision Language Models have demonstrated impressive versatile capabilities through extensive multimodal pre-training, but face significant limitations when incorporating specialized knowledge domains beyond their training distribution. These models struggle with a fundamental dilemma: direct adaptation approaches that inject domain-specific knowledge often trigger catastrophic forgetting of foundational visual-linguistic abilities. We introduce Structured Dialogue Fine-Tuning (SDFT), an effective approach that effectively injects domain-specific knowledge while minimizing catastrophic forgetting. Drawing inspiration from supervised fine-tuning in LLMs and subject-driven personalization in text-to-image diffusion models, our method employs a three-phase dialogue structure: Foundation Preservation reinforces pre-trained visual-linguistic alignment through caption tasks; Contrastive Disambiguation introduces carefully designed counterfactual examples to maintain semantic boundaries; and Knowledge Specialization embeds specialized information through chain-of-thought reasoning. Experimental results across multiple domains confirm SDFT's effectiveness in balancing specialized knowledge acquisition with general capability retention. Our key contributions include a data-centric dialogue template that balances foundational alignment with targeted knowledge integration, a weighted multi-turn supervision framework, and comprehensive evaluation across diverse knowledge types.
Paper Structure (33 sections, 5 equations, 3 figures, 6 tables)

This paper contains 33 sections, 5 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Personalization
  4. Personalized Large Vision Language Models
  5. Knowledge Injection in Language Models
  6. Method
  7. Preliminary
  8. Structured Dialogue Fine-Tuning
  9. Multi-turn Dialogue Architecture
  10. Dialogue Synthesis Process
  11. Weighted Multi-Turn Supervision
  12. Experiment Settings
  13. Dataset
  14. Specific Knowledge Injection Datasets
  15. General Capability Evaluation Datasets
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Structured multi-turn supervision enables knowledge injection without forgetting. The base LVLM (Qwen2-VL-2B) describes only surface-level content, failing to capture the deeper conceptual meaning (e.g., global warming). In contrast, the same model fine-tuned with our SDFT approach identifies the symbolic implications by linking visual elements to abstract concepts.
  • Figure 2: Overview of the SDFT framework. Given domain-specific images across diverse categories (personalized entities, abstract concepts, domain expertise), the framework constructs structured dialogues using a synthesis model. The dialogue triplets are used to fine-tune a pretrained LVLM with weighted cross-entropy loss coefficients that balance knowledge acquisition and general capability retention.
  • Figure 3: PCA visualization of hidden states when responding to target concepts (top) and unrelated concepts (bottom). Confidence ellipses (dashed lines) indicate distribution boundaries for each approach.