Markovian Generation Chains in Large Language Models

Abstract

The widespread use of large language models (LLMs) raises an important question: how do texts evolve when they are repeatedly processed by LLMs? In this paper, we define this iterative inference process as Markovian generation chains, where each step takes a specific prompt template and the previous output as input, without including any prior memory. In iterative rephrasing and round-trip translation experiments, the output either converges to a small recurrent set or continues to produce novel sentences over a finite horizon. Through sentence-level Markov chain modeling and analysis of simulated data, we show that iterative process can either increase or reduce sentence diversity depending on factors such as the temperature parameter and the initial input sentence. These results offer valuable insights into the dynamics of iterative LLM inference and their implications for multi-agent LLM systems.

Figures (12)

• Figure 1: Illustration of iterative LLM reprocessing (Markovian generation chains). Each node denotes the output after iteration $i$. Under greedy decoding, chains typically enter fixed points or short cycles, limiting sentence-level diversity. Under sampling-based decoding, stochasticity may yield longer transients and more distinct outputs
• Figure 2: Average number of unique paraphrases generated over 50 iterative rephrasings across three datasets, comparing four instruction-tuned LLMs: GPT-4o-mini, Llama-3.1-8B, Mistral-7B, and Qwen-2.5-7B. Results are shown for greedy decoding (orange) and sampling-based decoding (purple). Error bars represent one standard deviation.
• Figure 3: Evolution of text similarity metrics across 50 rephrasing iterations for the BookSum dataset using greedy decoding. Each iteration compares the current rephrased text against the previous iteration's text as reference.
• Figure 4: Pearson correlation $r$ between seed length (words) and the number of distinct outputs over $T=50$ iterations.
• Figure 5: Number of distinct sentences produced over 50 iterative rephrasings with GPT-4o-mini under different settings. P1 and P2 correspond to Listings \ref{['prompt_example']} and \ref{['prompt_ablation']} in the Appendix. Boxes denote the interquartile range, and the center line indicates the median.