Cognitively Layered Data Synthesis for Domain Adaptation of LLMs to Space Situational Awareness

TL;DR

BD-FDG (Bloom's Taxonomy-based Domain-specific Fine-tuning Data Generation) is proposed, a framework that addresses incomplete knowledge coverage, shallow cognitive depth, and limited quality controllability through three mechanisms: structured knowledge organization, cognitively layered question modeling, and automated quality control.

Abstract

Large language models (LLMs) demonstrate exceptional performance on general-purpose tasks. however, transferring them to complex engineering domains such as space situational awareness (SSA) remains challenging owing to insufficient structural alignment with mission chains, the absence of higher-order cognitive supervision, and poor correspondence between data quality criteria and engineering specifications. The core bottleneck is the construction of high-quality supervised fine-tuning (SFT) datasets. To this end, we propose BD-FDG (Bloom's Taxonomy-based Domain-specific Fine-tuning Data Generation), a framework that addresses incomplete knowledge coverage, shallow cognitive depth, and limited quality controllability through three mechanisms: structured knowledge organization, cognitively layered question modeling, and automated quality control. The framework uses a knowledge tree to ensure structured corpus coverage, designs a question generation scheme spanning nine categories and six cognitive levels from Remember to Create to produce samples with a continuous difficulty gradient, and applies a multidimensional scoring pipeline to enforce domain rigor and consistency. Using BD-FDG, we construct SSA-SFT, a domain dataset of approximately 230K samples, and fine-tune Qwen3-8B to obtain SSA-LLM-8B. Experiments show that SSA-LLM-8B achieves relative BLEU-1 improvements of 144\% (no-think) and 176\% (think) on the domain test set and a win rate of 82.21\% over the baseline in arena comparisons, while largely preserving general benchmark performance (MMLU-Pro, MATH-500). These results validate SFT data construction driven by cognitive layering as an effective paradigm for complex engineering domains and provide a transferable framework for domain-specific LLM adaptation.

Figures (3)

• Figure 1: Architecture of the BD-FDG framework. The domain knowledge base construction stage (left) parses mission chain literature into structured chunks and builds a hybrid dense--sparse retrieval index for multisource context assembly. The question generation stage (middle) applies nine Bloom's Taxonomy-guided prompt templates across six cognitive levels to produce questions with graduated difficulty from retrieved context. The SFT data synthesis stage (right) distills reasoning traces and answers from teacher models, scores each sample along four quality dimensions, and filters by a composite threshold to yield the final SSA-SFT dataset.
• Figure 2: Overview of the SSA-SFT and SSA-Test datasets. Left: three-tier mission chain knowledge system. Right: distribution of question types and sample counts.
• Figure 3: Two-dimensional ablation results. The central heatmap shows average quality scores for each $(\alpha, K)$ configuration; marginal bar plots depict the main effects of Top-$K$ (top) and $\alpha$ (right). The optimal configuration $(\alpha\!=\!0.50, K\!=\!5)$ attains a score of 7.55.