Understanding the Repeat Curse in Large Language Models from a Feature Perspective

TL;DR

The paper tackles the Repeat Curse in large language models by applying mechanistic interpretability with Sparse Autoencoders to identify Repetition Features. It introduces the Duplicatus Charm (DUC), a pipeline that locates repetition-prone layers via logit analysis and then steers specific SAE-derived features to induce or mitigate repetition. Across token- and paragraph-level repetitions in three models (GPT2-small, Gemma-2-2B, Llama-3.1-8B), repetition features cluster in intermediate/final layers and are often linked to names, time, and mathematics; deactivating these features reduces repetition with minimal impact on overall performance. The approach is validated with a custom repetition dataset and multiple metrics, showing that mitigation improves information-theoretic and perceptual signals (e.g., entropy), while maintaining perplexity. Overall, DUC provides a practical, interpretable route to reduce repetition in LLMs and offers insight into the underpinnings of the Repeat Curse.

Abstract

Large language models (LLMs) have made remarkable progress in various domains, yet they often suffer from repetitive text generation, a phenomenon we refer to as the "Repeat Curse". While previous studies have proposed decoding strategies to mitigate repetition, the underlying mechanism behind this issue remains insufficiently explored. In this work, we investigate the root causes of repetition in LLMs through the lens of mechanistic interpretability. Inspired by recent advances in Sparse Autoencoders (SAEs), which enable monosemantic feature extraction, we propose a novel approach, "Duplicatus Charm", to induce and analyze the Repeat Curse. Our method systematically identifies "Repetition Features" -the key model activations responsible for generating repetitive outputs. First, we locate the layers most involved in repetition through logit analysis. Next, we extract and stimulate relevant features using SAE-based activation manipulation. To validate our approach, we construct a repetition dataset covering token and paragraph level repetitions and introduce an evaluation pipeline to quantify the influence of identified repetition features. Furthermore, by deactivating these features, we have effectively mitigated the Repeat Curse. The source code of our work is publicly available at: https://github.com/kaustpradalab/repeat-curse-llm

Figures (10)

• Figure 1: Examples of Repeat Curse: (a) Token Repetition Scenario, (b) Paragraph Repetition Scenario.
• Figure 2: Illustration of our work (using GPT as an example). First line: The Repeat Curse is categorized into two scenarios: Token and Paragraph, and datasets are created accordingly. These datasets are used to evaluate and select the metrics. Second line: The identification of Repetition Features is divided into two steps: layer localization and feature localization. By identifying the repetition features, we can deactivate them to mitigate the Repeat Curse.
• Figure 3: Comparison of Metrics in Token Repetition Scenario (M=1)
• Figure 4: Illustration of the distribution of characteristics for repetition features identified two or more times across multiple datasets.
• Figure 5: Comparison of Metrics in Paragraph Repetition Scenario