Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Circuit Complexity of Hierarchical Knowledge Tracing and Implications for Log-Precision Transformers

Naiming Liu, Richard Baraniuk, Shashank Sonkar

Abstract

Knowledge tracing models mastery over interconnected concepts, often organized by prerequisites. We analyze hierarchical prerequisite propagation through a circuit-complexity lens to clarify what is provable about transformer-style computation on deep concept hierarchies. Using recent results that log-precision transformers lie in logspace-uniform $\mathsf{TC}^0$, we formalize prerequisite-tree tasks including recursive-majority mastery propagation. Unconditionally, recursive-majority propagation lies in $\mathsf{NC}^1$ via $O(\log n)$-depth bounded-fanin circuits, while separating it from uniform $\mathsf{TC}^0$ would require major progress on open lower bounds. Under a monotonicity restriction, we obtain an unconditional barrier: alternating ALL/ANY prerequisite trees yield a strict depth hierarchy for \emph{monotone} threshold circuits. Empirically, transformer encoders trained on recursive-majority trees converge to permutation-invariant shortcuts; explicit structure alone does not prevent this, but auxiliary supervision on intermediate subtrees elicits structure-dependent computation and achieves near-perfect accuracy at depths 3--4. These findings motivate structure-aware objectives and iterative mechanisms for prerequisite-sensitive knowledge tracing on deep hierarchies.

Circuit Complexity of Hierarchical Knowledge Tracing and Implications for Log-Precision Transformers

Abstract

Knowledge tracing models mastery over interconnected concepts, often organized by prerequisites. We analyze hierarchical prerequisite propagation through a circuit-complexity lens to clarify what is provable about transformer-style computation on deep concept hierarchies. Using recent results that log-precision transformers lie in logspace-uniform , we formalize prerequisite-tree tasks including recursive-majority mastery propagation. Unconditionally, recursive-majority propagation lies in via -depth bounded-fanin circuits, while separating it from uniform would require major progress on open lower bounds. Under a monotonicity restriction, we obtain an unconditional barrier: alternating ALL/ANY prerequisite trees yield a strict depth hierarchy for \emph{monotone} threshold circuits. Empirically, transformer encoders trained on recursive-majority trees converge to permutation-invariant shortcuts; explicit structure alone does not prevent this, but auxiliary supervision on intermediate subtrees elicits structure-dependent computation and achieves near-perfect accuracy at depths 3--4. These findings motivate structure-aware objectives and iterative mechanisms for prerequisite-sensitive knowledge tracing on deep hierarchies.
Paper Structure (39 sections, 5 theorems, 3 equations, 4 tables)

This paper contains 39 sections, 5 theorems, 3 equations, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theoretical Analysis of Transformer Computation
  4. Knowledge Tracing Methods and Prerequisite Structure
  5. Recursive and Hierarchical Computation in Neural Networks
  6. Background: Circuit Complexity and Transformers
  7. Prerequisite-Tree Knowledge Tracing: Upper Bound, Monotone Barrier, and an Open Question
  8. Definitions
  9. Concept tree and inputs.
  10. Majority prerequisite rule.
  11. Unconditional upper bound: $\mathrm{KT}_{\mathrm{MAJ}} \in \mathsf{NC}^1$
  12. Important caveat.
  13. A monotone barrier (unconditional): alternating prerequisite trees resist shallow monotone threshold circuits
  14. Why monotonicity is a meaningful restriction.
  15. Alternating ALL/ANY prerequisite trees.
  16. ...and 24 more sections

Key Result

theorem 1

For every fixed $k \ge 3$, $\mathrm{KT}_{\mathrm{MAJ}}$ on balanced $k$-ary trees with $n$ leaves is computable in $\mathsf{NC}^1$.

Theorems & Definitions (9)

  • theorem 1: Upper Bound
  • proof
  • theorem 2: Monotone threshold depth hierarchy via alternating prerequisite formulas
  • proof : Justification and references
  • lemma 1: AND/OR as restricted ternary majority
  • corollary 1: Monotone lower bound transfers to majority-tree KT
  • proof
  • corollary 2: Conditional transformer implication
  • proof