Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation

Tianyi Zhang, Florian Mai, Lucie Flek

TL;DR

IKnow tackles domain adaptation for instruction-tuned LLMs by avoiding reliance on base weights or external knowledge databases. It introduces a three-stage pipeline that converts unlabeled test contexts into instruction–response pairs and trains with three self-supervised tasks—Masked Token Prediction, Masked Phrase Prediction, and NL↔KG—to preserve instruction-following while encoding entity–relation semantics. Experimental results on RepliQA and SciQAG show consistent gains over next-token baselines, with notable improvements when employing knowledge-aware tasks, though effects vary by model. The work demonstrates a practical path to domain adaptation that leverages text-internal knowledge, with implications for safer, more robust deployment of instruction-tuned LLMs in new domains.

Abstract

Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level.

IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation

TL;DR

IKnow tackles domain adaptation for instruction-tuned LLMs by avoiding reliance on base weights or external knowledge databases. It introduces a three-stage pipeline that converts unlabeled test contexts into instruction–response pairs and trains with three self-supervised tasks—Masked Token Prediction, Masked Phrase Prediction, and NL↔KG—to preserve instruction-following while encoding entity–relation semantics. Experimental results on RepliQA and SciQAG show consistent gains over next-token baselines, with notable improvements when employing knowledge-aware tasks, though effects vary by model. The work demonstrates a practical path to domain adaptation that leverages text-internal knowledge, with implications for safer, more robust deployment of instruction-tuned LLMs in new domains.

Abstract

Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level.
Paper Structure (23 sections, 1 equation, 1 figure, 4 tables)

This paper contains 23 sections, 1 equation, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Continual pretraining.
  4. Test-time training and adaptation.
  5. Knowledge-intensive pretraining.
  6. Method
  7. Problem Setup
  8. Instruction-Knowledge-Aware Continual Adaptation (IKnow)
  9. Prepare data
  10. Masked Token Prediction
  11. Masked Phrase Prediction
  12. NL$\leftrightarrow$KG
  13. Experiments
  14. Experimental Setup
  15. Hypotheses and baselines
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Overview of our framework, consisting of three stages: (I)data preparation, (II)model pretraining, and (III)downstream evaluation. (I) In the data preparation stage, we leverage off-the-shelf syntactic parsers to extract structural information from text. Specifically, we employ two complementary strategies: (1) extracting constituency to identify phrases, and (2) deriving knowledge graphs based on dependency structures. (II) In the pretraining stage, we introduce two distinct objectives: Masked Phrase Prediction (MPP), which trains the model to reconstruct syntactically meaningful spans based on constituency structure, and the NL-KG Loop task, which encourages bidirectional reasoning between natural language and structured knowledge graphs with a forward pass NL2KG and a backward pass KG2NL. (III) Finally, we evaluate the resulting models—MPP-LLM and NLKG-Loop-LLM—on knowledge-intensive downstream tasks such as question answering, demonstrating the effectiveness of structure and semantic informed pretraining. All our tasks are knowledge-aware and formatted in the instruction-response template. The response is in underlined italic format following an arrow.