From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning

TL;DR

This work challenges the belief that finetuning inherently struggles to inject knowledge, showing that success depends on concrete factors such as information type, data format, and evaluation task. Through a large-scale grid search on Gemini v1.5 Pro/Flash, it demonstrates that question-answer formats are far more effective for knowledge retention than unstructured articles, while numerical facts are notably harder to retain. It also reveals that finetuning can robustly encode tonal styles, but knowledge injection often fails to transfer to multi-step reasoning, and the entity type (real, fictional, or persona) is not the main driver. The findings offer practical guidance for finetuning—a shift toward QA-style data for knowledge tasks and careful task-alignment considerations—while highlighting fundamental limits in parametric knowledge and reasoning.

Abstract

Finetuning provides a scalable and cost-effective means of customizing language models for specific tasks or response styles, with greater reliability than prompting or in-context learning. In contrast, the conventional wisdom is that injecting knowledge via finetuning results in brittle performance and poor generalization. We argue that the dichotomy of "task customization" (e.g., instruction tuning) and "knowledge injection" (e.g., teaching new facts) is a distinction without a difference. We instead identify concrete factors that explain the heterogeneous effectiveness observed with finetuning. To this end, we conduct a large-scale experimental study of finetuning the frontier Gemini v1.5 model family on a spectrum of datasets that are artificially engineered to interpolate between the strengths and failure modes of finetuning. Our findings indicate that question-answer training data formats provide much stronger knowledge generalization than document/article-style training data, numerical information can be harder for finetuning to retain than categorical information, and models struggle to apply finetuned knowledge during multi-step reasoning even when trained on similar examples -- all factors that render "knowledge injection" to be especially difficult, even after controlling for considerations like data augmentation and information volume. On the other hand, our findings also indicate that it is not fundamentally more difficult to finetune information about a real-world event than information about what a model's writing style should be.

Figures (11)

• Figure 3.1: (Left) Post-finetuning accuracy rates of Gemini v1.5 Pro in a Wikipedia knowledge injection experiment setting and a tone teaching experiment setting. The evaluation step of the latter involves having a verifier match 10 responses to 10 tones: task accuracy refers to the average proportion of responses correctly matched, "Exact" task accuracy refers to the empirical probability that all responses are correctly matched. (Right) Diagram of the tone teaching experiment: a chatbot exchange without inflection is rewritten in several tones and a third-party model is asked to match the exchanges to their tone.
• Figure 4.1: Heatmap of the accuracy of finetuned Gemini v1.5 Pro models across a variety of training data formats, evaluation tasks, and information types. Information quantity is fixed at 20 facts, and entity type is marginalized over real-world and fictional entities. Each cell reflects 10 random seeds.
• Figure 4.2: Heatmap of the accuracy of finetuned Gemini v1.5 Flash models across a variety of training data formats, evaluation tasks, and information types. Information quantity is fixed at 20 facts, and entity type is marginalized over real-world and fictional entities. Each cell reflects 10 random seeds.
• Figure 4.3: Plot of the accuracy of finetuned Gemini v1.5 Pro models across a variety of evaluation tasks, comparing reasoning-based training data versus the best training data format. Information quantity is fixed at 20 facts, entity type is real-world or fictional entity, and we marginalize over all information types.
• Figure 4.4: Plot of the accuracy of finetuned Gemini v1.5 Pro models across a variety of evaluation tasks, comparing Wikipedia-article-based training data versus the best training data format. Information quantity is fixed at 20 facts, entity type is real-world or fictional entity, and we marginalize over all information types.