Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data?

Yutong Yin, Zhaoran Wang

TL;DR

This work investigates whether Transformer models can perform compositional reasoning by connecting fragmented knowledge during training. Using the FTCT synthetic dataset, it shows that few-shot Chain-of-Thought prompting enables accurate assembly of long causal chains from fragmented training pieces, with zero-shot performance being limited. The paper argues that compositional reasoning emerges when training and testing data are sufficiently similar (quantified by $\lambda = M/N \geq 0.3$) and requires multi-layer attention, suggesting that Transformers learn an underlying generalizable program. It provides theoretical results showing a two-layer Transformer can simulate this program and offers empirical evidence via induction heads and attention-assignment analyses. Overall, the findings indicate Transformers can exhibit robust compositional reasoning under controlled conditions, shedding light on the mechanisms that enable generalizable reasoning in large language models.

Abstract

Humans exhibit remarkable compositional reasoning by integrating knowledge from various sources. For example, if someone learns ( B = f(A) ) from one source and ( C = g(B) ) from another, they can deduce ( C=g(B)=g(f(A)) ) even without encountering ( ABC ) together, showcasing the generalization ability of human intelligence. In this paper, we introduce a synthetic learning task, "FTCT" (Fragmented at Training, Chained at Testing), to validate the potential of Transformers in replicating this skill and interpret its inner mechanism. In the training phase, data consist of separated knowledge fragments from an overall causal graph. During testing, Transformers must infer complete causal graph traces by integrating these fragments. Our findings demonstrate that few-shot Chain-of-Thought prompting enables Transformers to perform compositional reasoning on FTCT by revealing correct combinations of fragments, even if such combinations were absent in the training data. Furthermore, the emergence of compositional reasoning ability is strongly correlated with the model complexity and training-testing data similarity. We propose, both theoretically and empirically, that Transformers learn an underlying generalizable program from training, enabling effective compositional reasoning during testing.

Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data?

TL;DR

This work investigates whether Transformer models can perform compositional reasoning by connecting fragmented knowledge during training. Using the FTCT synthetic dataset, it shows that few-shot Chain-of-Thought prompting enables accurate assembly of long causal chains from fragmented training pieces, with zero-shot performance being limited. The paper argues that compositional reasoning emerges when training and testing data are sufficiently similar (quantified by ) and requires multi-layer attention, suggesting that Transformers learn an underlying generalizable program. It provides theoretical results showing a two-layer Transformer can simulate this program and offers empirical evidence via induction heads and attention-assignment analyses. Overall, the findings indicate Transformers can exhibit robust compositional reasoning under controlled conditions, shedding light on the mechanisms that enable generalizable reasoning in large language models.

Abstract

Humans exhibit remarkable compositional reasoning by integrating knowledge from various sources. For example, if someone learns ( B = f(A) ) from one source and ( C = g(B) ) from another, they can deduce ( C=g(B)=g(f(A)) ) even without encountering ( ABC ) together, showcasing the generalization ability of human intelligence. In this paper, we introduce a synthetic learning task, "FTCT" (Fragmented at Training, Chained at Testing), to validate the potential of Transformers in replicating this skill and interpret its inner mechanism. In the training phase, data consist of separated knowledge fragments from an overall causal graph. During testing, Transformers must infer complete causal graph traces by integrating these fragments. Our findings demonstrate that few-shot Chain-of-Thought prompting enables Transformers to perform compositional reasoning on FTCT by revealing correct combinations of fragments, even if such combinations were absent in the training data. Furthermore, the emergence of compositional reasoning ability is strongly correlated with the model complexity and training-testing data similarity. We propose, both theoretically and empirically, that Transformers learn an underlying generalizable program from training, enabling effective compositional reasoning during testing.
Paper Structure (46 sections, 4 theorems, 49 equations, 13 figures, 4 tables, 1 algorithm)

This paper contains 46 sections, 4 theorems, 49 equations, 13 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Fragmented at Training, Chained at Testing
  4. Causal Strcture
  5. Data Generation
  6. Training and Testing Loss
  7. Empirical Findings
  8. Few-shot CoT Prompting Enables Compositional Reasoning
  9. The Similarity between Training and Testing Data Determines the Emgergence of Compositional Reasoning
  10. Multi-Layer Attention Mechanism Enables Compositional Reasoning
  11. Transformer Does Compositional Reasoning via the Underlying Program
  12. Underlying Program
  13. Transformer is Expressive Enough to Simulate the Underlying Program
  14. Empirical Evidence of the Underlying Program
  15. Induction Heads
  16. ...and 31 more sections

Key Result

Lemma 5.1

For any sentence $z_{1:T}:=\widetilde{\texttt{inp}}^k+\widetilde{\texttt{lab}}^k_{1:t}$, where input $\widetilde{\texttt{inp}}^k$ and label $\widetilde{\texttt{lab}}^k$ are sampled from ${\mathcal{D}}_{\text{train}}$ with $k\geq 1$, and $t$ is an arbitrary position within the label, denote the distr

Figures (13)

  • Figure 1: Overview of the FTCT Data Generation Process. The generation begins with the introduction of "Causal Structure", representing the relationships of knowledge points. The "Fragmented at Training" stage shows how shorter child chains with noise vertices are formed to simulate incomplete knowledge during training. The "Chained at Testing" stage presents the longest chains used in testing to assess compositional reasoning ability. The "Few-Shot Examples" part demonstrates the concatenation of multiple sequences for both training and testing to enable few-shot learning. In the end, the "Downstream Processing" adapts sequences into natural language-like sentences for intuitive reasoning format.
  • Figure 2: Left: Zero and few-shot testing performance of Transformers trained on FTCT with various causal depths and child chain lengths. Right: Relationship between the relative knowledge ratio and model's compositional reasoning ability. Each row shows testing performance under a specific criterion: The 1st row indicates whole chain accuracy (correct prediction of all vertices and values), the 2nd row shows testing vertices accuracy (correct order of vertices), and the 3rd row shows testing values accuracy (correct values of vertices with correct preceding reasoning paths).
  • Figure 3: Testing performance of Transformers trained on FTCT with all combinations of causal depths and child chain lengths. The performance is evaluated by all four criteria in Appendix \ref{['sec:all curves']}.
  • Figure 4: Testing performance of Transformers trained on FTCT where the testing data are blurred by noisy tokens. The performance is evaluated by all four criteria in Section \ref{['sec:test noise']}.
  • Figure 5: The relationship between the relative knowledge ratio and Transformers' compositional reasoning ability tested on the data blurred by noisy tokens.
  • ...and 8 more figures

Theorems & Definitions (4)

  • Lemma 5.1
  • Lemma 5.2
  • Lemma 5.3
  • Theorem 5.4