LLM Foundation Models: January 2026 Week 2

1. Transformer Is Inherently a Causal Learner

arXiv:2601.05647

Why Novel: Proves that decoder-only transformers trained for autoregressive forecasting inherently learn time-delayed causal structure in their representations—not as an emergent property but with formal identifiability guarantees.

Key Innovations:

• Score-gradient energy $H_{j,i}^{\ell} := \mathbb{E}[(\partial_{x_{j,t-\ell}} \log p(X_{i,t} \mid X_{<t}))^2]$ characterizes causal edges
• Layer-wise Relevance Propagation (LRP) aggregates gradient attributions to recover causal graphs
• Formal theorem: edge $j \stackrel{\ell}{\longrightarrow} i$ exists iff $H_{j,i}^{\ell} > 0$ under standard assumptions
• Robust to latent confounders, non-stationarity, and nonlinear dynamics

Evidence:

• thm:1 — Main identifiability theorem: lagged causal graph $\mathcal{G}^*$ uniquely recoverable via score gradient energy
• fig:1 — Data generation and transformer-based causal discovery framework
• fig:2 — F1 score analysis across high-dimensional, long-range, nonlinear, and non-stationary regimes
• tab:1 — Benchmark results: DOT achieves competitive AUROC/AUPRC across AQI, Traffic, Medical datasets

Impact: Transforms causal discovery from specialized algorithms to standard next-token prediction. Any pretrained autoregressive model carries causal structure extractable via gradient analysis.

2. The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoning

arXiv:2601.06002

Why Novel: Reframes Long CoT reasoning as a macromolecular structure governed by three bond-like interactions, providing a principled framework for understanding why distillation from weak models fails while strong reasoning LLMs succeed.

Key Innovations:

• Three reasoning bonds: Deep Reasoning (covalent), Self-Reflection (hydrogen), Self-Exploration (van der Waals)
• Semantic Transfer Graph shows stable cross-model reasoning patterns (Pearson >0.9)
• Mole-Syn: synthesizes molecular structure from weak instruction LLMs to match reasoning LLM distillation
• Demonstrates ICL and human-annotated traces fail to acquire stable Long CoT structure

Evidence:

• fig:1 — Molecular structure hypothesis: three chemical bonds governing Long CoT stability
• fig:3 — Failure of distillation from weak instruction LLMs vs success from strong reasoning LLMs
• fig:5 — Transfer graph stability across Llama/Qwen models (Pearson >0.95 at 2000+ samples)
• tab:1 — OSS-Distill-Data achieves 39.27% avg vs 25.32% baseline on 6 math benchmarks

Impact: Provides first principled explanation for Long CoT distillation dynamics. Opens path to synthesizing reasoning structure without access to frontier reasoning models.

3. Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer

arXiv:2601.05770

Why Novel: Bridges continuous neural representations and discrete symbolic logic by design, enabling extraction of executable Python algorithms from trained Transformer models with provable correctness.

Key Innovations:

• Discrete Transformer: Numerical Attention for routing, Numerical MLP for arithmetic with functional disentanglement
• Temperature-annealed sampling yields interpretable primitives (fixed offsets, windowed extrema, symbolic expressions)
• Hypothesis testing identifies attention patterns; symbolic regression approximates MLP transformations
• Successfully extracts algorithms for parity, max/min, bitwise ops, and physics simulations

Evidence:

• fig:1 — Framework overview: Discrete Search → Algorithm Extraction → Synthesized Python Program
• tab:1 — Near-zero MSE loss across 16 algorithm tasks including physics simulations
• fig:3 — Extracted parity_last2 code with symbolic simplification $y_t = x_t + x_{t-1} - 2x_tx_{t-1}$
• fig:5 — Training dynamics: loss decreases early, discretization Agreement→1.0 during temperature annealing

Impact: Makes interpretability a design choice rather than post-hoc analysis. Enables mechanistic verification of learned algorithms.

1. PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning

arXiv:2601.05593

Breaks sequential reasoning bottleneck via massive parallel exploration coordinated through learned coordination mechanisms.

2. Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space

arXiv:2601.06220

ZeroRouter enables zero-shot onboarding of new LLMs via context-aware latent predictor and D-optimal anchor profiling.

3. ConMax: Confidence-Maximizing Compression for Efficient Chain-of-Thought Reasoning

arXiv:2601.04973

RL-based compression achieves 43% CoT length reduction with 0.7% accuracy loss via dual confidence rewards.

4. Token-Level LLM Collaboration via FusionRoute

arXiv:2601.05106

Joint expert selection and corrective logit fusion at each decoding step with theoretical token-level routing analysis.

5. SPINAL -- Scaling-law and Preference Integration in Neural Alignment Layers

arXiv:2601.06238

Diagnostic measuring how alignment signal distributes across layers, enabling checkpoint comparison and failure prediction.