Distilling Rule-based Knowledge into Large Language Models

TL;DR

The paper tackles the challenge of encoding rule-based knowledge into large language models beyond sample-based learning. It introduces rule distillation, a framework that leverages in-context learning to extract rule signals from textual instructions and then encodes these rules into model parameters by jointly distilling output distributions and hidden-state representations. Empirical results across arithmetic, safety, and sentiment-steering tasks show that rule distillation, especially with hidden-state alignment, yields faster generalization and stronger adherence to rule-driven behavior than instruction tuning with examples. This approach promises more efficient rule-based learning in LLMs and highlights practical considerations and limitations for broader, multi-rule applications in real-world tasks.

Abstract

Large language models (LLMs) have shown incredible performance in completing various real-world tasks. The current paradigm of knowledge learning for LLMs is mainly based on learning from examples, in which LLMs learn the internal rule implicitly from a certain number of supervised examples. However, this learning paradigm may not well learn those complicated rules, especially when the training examples are limited. We are inspired that humans can learn the new tasks or knowledge in another way by learning from rules. That is, humans can learn new tasks or grasp new knowledge quickly and generalize well given only a detailed rule and a few optional examples. Therefore, in this paper, we aim to explore the feasibility of this new learning paradigm, which targets on encoding rule-based knowledge into LLMs. We further propose rule distillation, which first uses the strong in-context abilities of LLMs to extract the knowledge from the textual rules, and then explicitly encode the knowledge into the parameters of LLMs by learning from the above in-context signals produced inside the model. Our experiments show that making LLMs learn from rules by our method is much more efficient than example-based learning in both the sample size and generalization ability. Warning: This paper may contain examples with offensive content.

Figures (4)

• Figure 1: Illustrations of our rule distillation approach and the tasks used in our experiments. Current learning paradigm mainly makes the LLM learn from examples; while we aim to enable the LLM to learn from rules and generalize the learned rules to all related inputs. We achieve this by aligning the hidden and output distributions of the target LLM on task examples only with the hidden and output distributions produced by a base LLM when it is performing in-context learning on both the task examples and task rules
• Figure 2: The results on the arithmetic task. Our proposed rule distillation method achieves consistently better performance on both the base and generalization sets under various few-shot settings than instruction tuning.
• Figure 3: The full results with different sizes of models on the base set of the arithmetic task.
• Figure 4: Evaluation curves on the safety and sentiment tasks under different numbers of training samples.