Partial Answer of How Transformers Learn Automata
Tiantian Zhang
TL;DR
The paper tackles the question of which natural automata admit extremely shallow Transformer implementations, aiming for depth independent of both input length $T$ and state size $|Q|$. It develops a representation-theoretic framework that uses Fourier modules to embed automaton states (e.g., $Q= Z_p$) and to realize transitions as frequency-domain operations, yielding constant-depth simulations. The approach generalizes beyond abelian groups to finite semigroups and non-abelian groups via an embedding into direct sums of irreducible representations, with depth bounds $O(1)$ for semigroup automata and $O( ) for semidirect-product automata, at widths tied to representation sizes $D= _j d_j^2$. It further extends to finite monoids and semidirect-product structures $N times H$, showing that Transformers can simulate such automata with $O( )$ depth and widths $O((N)+( ho_H))$, thus going beyond Krohn–Rhodes decompositions. The results include discussions of solvable versus unsolvable groups, where depth remains constant while width grows with representation dimensions, and provide an open-question answer illustrating a broad class of automata amenable to efficient Transformer simulations with practical implications for frequency-domain reasoning and structured memory handling.
Abstract
We introduce a novel framework for simulating finite automata using representation-theoretic semidirect products and Fourier modules, achieving more efficient Transformer-based implementations.