Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding

Ruohao Guo, Wei Xu, Alan Ritter

TL;DR

This work targets cross-style generalization in language style understanding by introducing lexicon-based instructions and meta-tuning of LLMs on source styles. The method leverages style lexicons to elicit latent stylistic knowledge, with class randomization to prevent memorization and promote genuine lexicon use. A benchmark of 13 established styles plus 63 LL-generated styles demonstrates consistent zero-shot gains across multiple models, and the approach improves few-shot stability by reducing sensitivity to example selection. The combined use of lexicons and meta-learning advances the practical ability to analyze and generate stylistic text, with potential applications in toxicity detection, readability assessment, and nuanced language analysis, while acknowledging limitations around lexicon quality, language scope, and dataset biases.

Abstract

Language style is often used by writers to convey their intentions, identities, and mastery of language. In this paper, we show that current large language models struggle to capture some language styles without fine-tuning. To address this challenge, we investigate whether LLMs can be meta-trained based on representative lexicons to recognize new styles they have not been fine-tuned on. Experiments on 13 established style classification tasks, as well as 63 novel tasks generated using LLMs, demonstrate that meta-training with style lexicons consistently improves zero-shot transfer across styles. We release the code and data at http://github.com/octaviaguo/Style-LLM .

Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding

TL;DR

This work targets cross-style generalization in language style understanding by introducing lexicon-based instructions and meta-tuning of LLMs on source styles. The method leverages style lexicons to elicit latent stylistic knowledge, with class randomization to prevent memorization and promote genuine lexicon use. A benchmark of 13 established styles plus 63 LL-generated styles demonstrates consistent zero-shot gains across multiple models, and the approach improves few-shot stability by reducing sensitivity to example selection. The combined use of lexicons and meta-learning advances the practical ability to analyze and generate stylistic text, with potential applications in toxicity detection, readability assessment, and nuanced language analysis, while acknowledging limitations around lexicon quality, language scope, and dataset biases.

Abstract

Language style is often used by writers to convey their intentions, identities, and mastery of language. In this paper, we show that current large language models struggle to capture some language styles without fine-tuning. To address this challenge, we investigate whether LLMs can be meta-trained based on representative lexicons to recognize new styles they have not been fine-tuned on. Experiments on 13 established style classification tasks, as well as 63 novel tasks generated using LLMs, demonstrate that meta-training with style lexicons consistently improves zero-shot transfer across styles. We release the code and data at http://github.com/octaviaguo/Style-LLM .
Paper Structure (59 sections, 9 figures, 20 tables)

This paper contains 59 sections, 9 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Meta-Tuning for Style Generalization
  3. Problem Definition and Approach
  4. Meta-Tuning on Source Styles.
  5. Zero-Shot Evaluation on Unseen Target Styles.
  6. A Benchmark for Style Generalization
  7. Style Datasets.
  8. Lexicon Collection.
  9. Experimental Settings
  10. Zero-Shot
  11. Few-Shot
  12. MetaICL min-etal-2022-metaiclflant5.
  13. MetaICL+Lex.
  14. Results and Analysis
  15. Pre-trained Language Models
  16. ...and 44 more sections

Figures (9)

  • Figure 1: Overview of using lexicon-based instructions for cross-style zero-shot classification. It consists of two steps: (1) instruction tuning the model on training styles; (2) evaluating the learned model on unseen target styles zero-shot. A lexicon-based instruction is composed of instruction, class names, lexicons and an input.
  • Figure 2: Zero-shot performance when fine-tuning with different lexicon-based instruction variants. Instruction tuning with class Randomization shows advantages over those without. Instructions with natural language perform generally better than those without. We also compare fine-tuning with generic identifiers, a method that differs from the "Lang" variant by mapping class names of each style to a fixed set of generic names (e.g., Style A, Style B). This approach improves model generalization over "Lang", but generally falls short of the performance achieved with our optimal randomized identifiers "Lang, Rw".
  • Figure 3: Examples of ChatGPT output for different style classes.
  • Figure 4: Distribution of 63 style classification tasks in §\ref{['sec:gpt4-styles']}.
  • Figure 5: Examples of different lexicon-based instruction variants (as detailed in §\ref{['variant_lps']}) on Politeness. Red part is (randomized) classes, the green part represents the words sampled from each class lexicon, and yellow stands for the input sentence and the uncolored part is the instruction template.
  • ...and 4 more figures