Neural Induction of Finite-State Transducers

TL;DR

This work introduces a pipeline to automatically induce unweighted finite-state transducers (FSTs) from recurrent neural networks by capitalizing on the geometry of hidden states. The method aligns input-output pairs with CRPAlign, trains a one-layer Elman RNN with a transduction objective augmented by a spectral-norm penalty, collects hidden activations (including synthetic data to cover the input domain), clusters activations into FST states, and applies a state-splitting/minimization process to enforce determinism. Across morphological inflection, grapheme-to-phoneme, and historical normalization tasks, the extracted transducers achieve high accuracy and generally outperform traditional OSTIA-based baselines, approaching or matching human-expert performance in some cases. The results demonstrate the practicality of deriving high-performance symbolic transducers from neural models and highlight avenues for extending the approach with bidirectional architectures or transformers to handle more complex dependencies more effectively.

Abstract

Finite-State Transducers (FSTs) are effective models for string-to-string rewriting tasks, often providing the efficiency necessary for high-performance applications, but constructing transducers by hand is difficult. In this work, we propose a novel method for automatically constructing unweighted FSTs following the hidden state geometry learned by a recurrent neural network. We evaluate our methods on real-world datasets for morphological inflection, grapheme-to-phoneme prediction, and historical normalization, showing that the constructed FSTs are highly accurate and robust for many datasets, substantially outperforming classical transducer learning algorithms by up to 87% accuracy on held-out test sets.

Figures (8)

• Figure 1: There is a theoretical and empirical correspondence between recurrent neural network's continuous state space (left) and the finite-state space of an automaton (right). In the RNN state space, individual activation values may form clusters that correspond to finite states in the FSA (blue). Transitions between individual activations can be aggregated to form the transitions of the FSA (purple).
• Figure 2: Example FST that rewrites "cat" to "cats". The input and output alphabet are combined.
• Figure 3: State splitting algorithm, where the minimum threshold $\lambda_{trans} = 2$. If a cluster has multiple possible transitions each over the threshold (1), the points are split using an SVM or logistic regression (2). If there are still conflicting transitions, but one of them is under the threshold, it is removed (3). Finally, clusters are materialized into FST states (4).
• Figure 4: Scores with CRPAlign algorithm and simple minimum-edit distance (MED).
• Figure 5: Scores with and without synthetic data.