Exploring Model Editing for LLM-based Aspect-Based Sentiment Classification

TL;DR

This work addresses efficient adaptation of LLMs to aspect-based sentiment classification (ABSC) by linking causal tracing with targeted model edits. It identifies mid-layer representations of aspect terms as the critical components for ABSC decisions and proposes two editing strategies—weight-based editing of attention projections and representation-based editing of residual streams—to modify only a small subset of parameters. Through in-domain and out-of-domain experiments on Llama2-7b, the approach achieves competitive accuracy with far fewer trainable parameters than full fine-tuning and several PEFT methods, while providing interpretable insights into which components drive ABSC. The findings highlight the practicality of combining causal interventions with targeted edits to enable efficient, robust ABSC across domains, with potential applicability to other LLM-tuned tasks.

Abstract

Model editing aims at selectively updating a small subset of a neural model's parameters with an interpretable strategy to achieve desired modifications. It can significantly reduce computational costs to adapt to large language models (LLMs). Given its ability to precisely target critical components within LLMs, model editing shows great potential for efficient fine-tuning applications. In this work, we investigate model editing to serve an efficient method for adapting LLMs to solve aspect-based sentiment classification. Through causal interventions, we trace and determine which neuron hidden states are essential for the prediction of the model. By performing interventions and restorations on each component of an LLM, we identify the importance of these components for aspect-based sentiment classification. Our findings reveal that a distinct set of mid-layer representations is essential for detecting the sentiment polarity of given aspect words. Leveraging these insights, we develop a model editing approach that focuses exclusively on these critical parts of the LLM, leading to a more efficient method for adapting LLMs. Our in-domain and out-of-domain experiments demonstrate that this approach achieves competitive results compared to the currently strongest methods with significantly fewer trainable parameters, highlighting a more efficient and interpretable fine-tuning strategy.

Figures (5)

• Figure 1: An illustration of our 2-steps model editing method for LLM-based aspect-based sentiment classification.
• Figure 2: Illustration of the process of causal tracing and model editing for ABSC. (a), (b), (c) illustrate three runs for tracing sentiment associations; (d) and (e) demonstrate our method, where colored blocks represent active parameters and the grey blocks represent frozen parameters. Notably, representation-based editing is applied only to the position of the aspect word.
• Figure 3: Causal tracing results for ABSC at various words, layers, and positions. The "First token," "Second token," and so on denote the positions of the input sentence tokens.
• Figure 4: Impact on trainable layers of LoReft and our method.
• Figure 5: Influence on specific positions: 'aspect' refers to editing on aspect terms, 'last' refers to editing on the last words, and 'mid' refers to editing on a random mid-position word.