Learnable Game-theoretic Policy Optimization for Data-centric Self-explanation Rationalization

TL;DR

Data-centric rationalization (RNP) enables explanations by a generator selecting a rationale $\hat{Z}$ for which a predictor yields $\hat{Y}$, but training often converges to a suboptimal equilibrium due to mode collapse. The authors model the interaction as a two-agent game and propose PORAT, a policy-optimization framework that progressively intervenes in the generator and predictor to restore exploration when the critic’s value estimate incurs an error $\epsilon(s)$, i.e., $V^\pi(s)=V^*(s)-\epsilon(s)$, making $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)$ nonzero. They prove feasibility and provide an algorithm with steps such as temporarily freezing the predictor and setting $V^\pi(s^{t+1})=0$, coupled with a multi-time-scale update $N$, validated on nine real-world datasets and two synthetic settings, achieving up to $8.1\%$ improvement in $F1$. The results show improved faithfulness and diversity of generated rationales while maintaining accuracy, suggesting broader applicability to data-centric explainability and potential integration with large language model–based rationalization.

Abstract

Rationalization, a data-centric framework, aims to build self-explanatory models to explain the prediction outcome by generating a subset of human-intelligible pieces of the input data. It involves a cooperative game model where a generator generates the most human-intelligible parts of the input (i.e., rationales), followed by a predictor that makes predictions based on these generated rationales. Conventional rationalization methods typically impose constraints via regularization terms to calibrate or penalize undesired generation. However, these methods are suffering from a problem called mode collapse, in which the predictor produces correct predictions yet the generator consistently outputs rationales with collapsed patterns. Moreover, existing studies are typically designed separately for specific collapsed patterns, lacking a unified consideration. In this paper, we systematically revisit cooperative rationalization from a novel game-theoretic perspective and identify the fundamental cause of this problem: the generator no longer tends to explore new strategies to uncover informative rationales, ultimately leading the system to converge to a suboptimal game equilibrium (correct predictions v.s collapsed rationales). To solve this problem, we then propose a novel approach, Game-theoretic Policy Optimization oriented RATionalization (PORAT), which progressively introduces policy interventions to address the game equilibrium in the cooperative game process, thereby guiding the model toward a more optimal solution state. We theoretically analyse the cause of such a suboptimal equilibrium and prove the feasibility of the proposed method. Furthermore, we validate our method on nine widely used real-world datasets and two synthetic settings, where PORAT achieves up to 8.1% performance improvements over existing state-of-the-art methods.

Figures (7)

• Figure 1: A standard RNP framework on the binary sentiment analysis, where $X$, $Z$, $\hat{Y}$, $Y$ represent the input data, rationale, prediction and the groundtruth label, respectively.
• Figure 2: Rationales of different patterns illustrate the rationales of rational, spurious correlation, partial degeneration, and complete degeneration, which are caused by the generator. Human-annotated rationales are underlined; “red text” indicates rationales generated by models.
• Figure 3: The coordination mechanism in RNP framework.
• Figure 4: The overview architecture of PORAT, where $X,Z,\hat{Y},Y$ represent the input text, rationale, prediction and the groundtruth label, respectively. Here, we provide policy intervention at $(t+1)$-th timestep, which is a progressive policy optimization to help the whole model escape a suboptimal state.
• Figure 5: Ablation Studies on (a-c) BeerAdvocate beer, (d-f) BeerAdvocate* lei2016rnp and (g-i) HotelReview hotel Benchmarks.

Theorems & Definitions (7)

• Definition 1: Two-Agent Markov Games for Rationalization
• Definition 2: Game-theoretic Policy for Rationalization
• Proposition 1: Dependence and Non-discriminatory
• Lemma 1
• Theorem 1
• Lemma 2
• Theorem 2