IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neurons

TL;DR

This work proposes a novel framework, IRCAN (Identifying and Reweighting Context-Aware Neurons), to capitalize on neurons that are crucial in processing contextual cues, and offers a scalable, plug-and-play solution that can be integrated seamlessly with existing models.

Abstract

It is widely acknowledged that large language models (LLMs) encode a vast reservoir of knowledge after being trained on mass data. Recent studies disclose knowledge conflicts in LLM generation, wherein outdated or incorrect parametric knowledge (i.e., encoded knowledge) contradicts new knowledge provided in the context. To mitigate such knowledge conflicts, we propose a novel framework, IRCAN (Identifying and Reweighting Context-Aware Neurons) to capitalize on neurons that are crucial in processing contextual cues. Specifically, IRCAN first identifies neurons that significantly contribute to context processing, utilizing a context-aware attribution score derived from integrated gradients. Subsequently, the identified context-aware neurons are strengthened via reweighting. In doing so, we steer LLMs to generate context-sensitive outputs with respect to the new knowledge provided in the context. Extensive experiments conducted across a variety of models and tasks demonstrate that IRCAN not only achieves remarkable improvements in handling knowledge conflicts but also offers a scalable, plug-and-play solution that can be integrated seamlessly with existing models. Our codes are released at https://github.com/danshi777/IRCAN.

Figures (13)

• Figure 1: The diagram of IRCAN. When an LLM faces a knowledge conflict between the context and its inherent knowledge, IRCAN first calculates the attribution score for each neuron to measure its contribution to processing the context. It then identifies context-aware neurons by taking the intersection of neurons with the highest scores. Subsequently, the identified neurons are reweighted so that IRCAN could guide the model to be more aligned with the contextual knowledge, ensuring greater fidelity to the context.
• Figure 2: The results of ablation studies to illustrate the accuracy implications of different interventions. ErCAN denotes the variant where context-aware neurons are erased. ERN represents the enhancement of random neurons. ErRN indicates the erasure of random neurons.
• Figure 3: Model performance with different enhancement strengths $\beta$.
• Figure 4: Model performance with different enhanced # neurons $h$.
• Figure 5: The distribution of context-aware neurons across layers with various LLMs.