KBAlign: Efficient Self Adaptation on Specific Knowledge Bases

TL;DR

KBAlign presents a self-supervised framework for domain-specific KBQA that obviates external supervision by using multi-grained self-annotation and iterative self-verification to align models with small textual KBs. It couples unsupervised tuning with targeted inference (e.g., query expansion) to improve retrieval-augmented generation without large data or external signals. Across four datasets and multiple backbones, KBAlign achieves substantial QA improvements and approaches GPT-4–supervised adaptation performance at a fraction of the cost, with detailed ablations guiding practical deployment. The work highlights practical trade-offs, including the importance of long-dependency annotation for certain tasks and the need to consider retriever adaptation for further gains.

Abstract

Although retrieval-augmented generation (RAG) remains essential for knowledge-based question answering (KBQA), current paradigms face critical challenges under specific domains. Existing methods struggle with targeted adaptation on small-scale KBs: vanilla unsupervised training exhibits poor effectiveness, while fine-tuning incurs prohibitive costs of external signals. We present KBAlign, a self-supervised framework that enhances RAG systems through efficient model adaptation. Our key insight is to leverage the model's intrinsic capabilities for knowledge alignment through two innovative mechanisms: multi-grained self-annotation that captures global knowledge for data construction, and iterative tuning that accelerates convergence through self verification. This framework enables cost-effective model adaptation to specific textual KBs, without human supervision or external model assistance. Experiments demonstrate that KBAlign can achieve 90\% of the performance gain obtained through GPT-4-supervised adaptation, while relying entirely on self-annotation of much smaller models. KBAlign significantly improves downstream QA accuracy across multiple domains with tiny costs, particularly benefiting scenarios requiring deep knowledge integration from specialized corpora. We release our experimental data, models, and process analyses to the community for further exploration (https://github.com/thunlp/KBAlign).

Figures (6)

• Figure 1: KBAlign schematic. We design special self-annotation methods to help master global KB knowledge, conduct iterative verifying to save training time costs, and adopt targeted inference to improve accuracy.
• Figure 2: (a) Details for the KBAlign framework; (b) Instances for different annotation strategies and tasks.
• Figure 3: The impact of training amount on LooGLE performance. 'w GPT' refers to training with GPT-annotated data.
• Figure 4: The impact of iteration times and data amount for fixed training steps on LooGLE performance.
• Figure 5: Cases for KBAlign and baseline comparison. We display the translation for the Chinese JEC-QA task. The bold text and underlined text providing correct and wrong information for the QA process.