Action Controlled Paraphrasing

TL;DR

This work tackles paraphrase generation with user-friendly, action-based control to overcome the limitations of syntax-template or exemplar-based approaches. It introduces action tokens $K$, $P$, and an optional $O$, with action embeddings fused into text representations via a self-attention encoder, and a placeholder action token to bridge training and inference gaps. The authors evaluate both specific and optional action control on QQP and Twitter, demonstrating that action-guided generation can achieve precise paraphrasing when actions are provided and robust performance even when actions are omitted, aided by inference-alignment training. The approach advances user-centered paraphrasing and suggests extensions to lexical substitution and adversarial paraphrase applications, with implications for broader controllable generation tasks. Overall, the method offers a practical, end-to-end framework for action-driven paraphrase synthesis and analysis of how user intent shapes output quality and diversity.

Abstract

Recent studies have demonstrated the potential to control paraphrase generation, such as through syntax, which has broad applications in various downstream tasks. However, these methods often require detailed parse trees or syntactic exemplars, countering human-like paraphrasing behavior in language use. Furthermore, an inference gap exists, as control specifications are only available during training but not during inference. In this work, we propose a new setup for controlled paraphrase generation. Specifically, we represent user intent as action tokens, embedding and concatenating them with text embeddings, thus flowing together into a self-attention encoder for representation fusion. To address the inference gap, we introduce an optional action token as a placeholder that encourages the model to determine the appropriate action independently when users' intended actions are not provided. Experimental results show that our method successfully enables precise action-controlled paraphrasing and preserves or even enhances performance compared to conventional uncontrolled methods when actions are not given. Our findings promote the concept of action-controlled paraphrasing for a more user-centered design.

Figures (4)

• Figure 1: An example from Quora Question Pair dataset showing the high-level overview of action controlled paraphrasing. Given the same text $\mathbf{x}$, the goal is to generate different paraphrase $\mathbf{y}_{1}$ and $\mathbf{y}_{2}$ conditioned on the corresponding actions $\mathbf{a}_{1}$ and $\mathbf{a}_{2}$. It is supposed to keep words in blue but to paraphrase those in red.
• Figure 2: (a) explains the proposed training strategy with specific weights. Each action is represented as a sequence in colors, where $K$ means to keep, red $P$ means to paraphrase, and green $O$ denotes a special action $O$ for inference alignment; (b) illustrates action embedding in the training pipeline. The encoder input is the concatenation of three states, that is, the input embedding of $\mathbf{x}$, the corresponding positional encoding, and the action embedding of $\mathbf{z}$. In practice, "[CLS]" is marked as $P$ to remove and "[SEP]" is marked as $K$ to reserve.
• Figure 3: The cross-attention dependencies of the second decoder layer when generating the token "are" and "can" for paraphrases $\mathbf{y^{\prime}_{1}}$ (left) and $\mathbf{y^{\prime}_{2}}$ (right) of the first case in Table \ref{['table:case']}. The colors of each column represent the distribution of an attention head.
• Figure 4: Evaluation of TFM* on Quora1 with the optional action control. Two metrics, iBLEU-0.8 (left) and BLEU-4 (right), are reported against different action weights standing for the proportion of keeping the true action or replacing it with an optional action. The green dashed line is the baseline TFM.