RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles

TL;DR

This paper introduces RECALL, a mechanism where self-referencing causal cycles created by cycle tokens allow autoregressive language models to recall preceding tokens from succeeding ones, addressing the reversal curse. It provides a formal probabilistic framework showing why left-context is hard to recover from right-context in standard autoregression and how cycle tokens create a loop that enables bidirectional influence. The authors validate the concept through deterministic and stochastic toy experiments, as well as observations in pretraining corpora, demonstrating that cycle tokens facilitate reliable recall and can scale to larger models with appropriate prompting. A two-step ReCall-aware prompting strategy is proposed to resolve the preceding-line problem in famous texts, achieving strong performance on key writings, suggesting practical implications for improving memory retrieval in LLMs while acknowledging limitations related to interpretability and deployment in retrieval-augmented setups.

Abstract

We introduce the concept of the self-referencing causal cycle (abbreviated RECALL) - a mechanism that enables large language models (LLMs) to bypass the limitations of unidirectional causality, which underlies a phenomenon known as the reversal curse. When an LLM is prompted with sequential data, it often fails to recall preceding context. For example, when we ask an LLM to recall the line preceding "O say does that star-spangled banner yet wave" in the U.S. National Anthem, it often fails to correctly return "Gave proof through the night that our flag was still there" - this is due to the reversal curse. It occurs because language models such as ChatGPT and Llama generate text based on preceding tokens, requiring facts to be learned and reproduced in a consistent token order. While the reversal curse is often viewed as a limitation, we offer evidence of an alternative view: it is not always an obstacle in practice. We find that RECALL is driven by what we designate as cycle tokens - sequences that connect different parts of the training data, enabling recall of preceding tokens from succeeding ones. Through rigorous probabilistic formalization and controlled experiments, we demonstrate how the cycles they induce influence a model's ability to reproduce information. To facilitate reproducibility, we provide our code and experimental details at https://anonymous.4open.science/r/remember-B0B8/.

Figures (11)

• Figure 1: In a text containing the U.S. National Anthem, the phrase "Star-Spangled Banner" appears several times and functions as a cycle token-sequence, creating a loop termed a self-referencing causal cycle. This cycle reconnects later parts of the text to earlier ones. Without these cycle tokens, the model would be unable to predict the preceding line "O! thus be it ever, when freemen shall stand" from the succeeding line "Between their lov'd home and the war's desolation" due to left-to-right causality (the red line). The blue line indicates that the cycle token-sequence occurs at multiple points in the text, acting as a reference point for the model to retrieve preceding context.
• Figure 2: A token sequence $[e1, e2, e3]$ illustrates the difficulty of predicting $e2$ from $e3$ due to left-to-right causality. By appending token $e1$ to the end of the sequence, a loop is formed, allowing the model to transition from $e3$ back to $e2$. Red and blue indicate the same concepts as in Figure \ref{['cycle_diagram_text']}.
• Figure 3: Performance of cycle tokens in few-token experiments, showing validation accuracy as a function of optimization steps (epochs). Train-validation splits follow Figure \ref{['table_for_few_token']}. Cycle tokens enable transitions from the starting to the target token-sequence, bypassing left-to-right causality. Last accuracy refers to correct prediction of the final token-sequence---the base case of which is a single token---while All accuracy refers to correct prediction of all subsequent tokens after the first. For example, in a validation sequence $(e_3, e_1, e_2)$, All means correctly predicting $e_1$ and $e_2$, while Last refers to predicting $e_2$. In a mixed-sequence case like $(E_3, e_1, E_2)$, All requires correctly predicting both $e_1$ and $E_2$, and Last refers to the correct prediction of $E_2$. In all scenarios, 100% accuracy is achieved in predicting the left-hand sequence using the cycle token as a bridge.
• Figure 4: Increasing the value of $\mathcal{N}$ results in longer token-sequences, but validation accuracy remains high with sufficient embedding dimensionality.
• Figure 5: Accuracy curves for four stochastic few-token tasks: The left column shows token-level tasks, and the right column shows sequence-level counterparts at candidates size $n=3$. The curves show accuracy on both the final token and all tokens, demonstrating the model’s ability to resolve specific target tokens despite stochastic variation during training.