SymSeqBench: a unified framework for the generation and analysis of rule-based symbolic sequences and datasets

TL;DR

SymSeqBench presents a domain-agnostic framework for generating and analyzing structured symbolic sequences rooted in Formal Language Theory. It couples SymSeq for grammar-based sequence generation with SeqBench for grounding symbols into embeddings and datasets, enabling multi-scale analysis from token to grammar levels and complexity-controlled task design such as AGL and NAD paradigms. A key contribution is complexity-guided grammar synthesis using Topological Entropy $TE = \log(\lambda_1)$ and ERGM-based sampling to produce regular grammars with target complexity, plus a comprehensive, modular metric suite that supports inference and benchmarking across biological, neuromorphic, and artificial systems. The framework facilitates reproducible cross-domain research, bridging cognitive science and AI by providing open-source tools, YAML-driven pipelines, and flexible embedding strategies for experiment-ready data and scalable benchmarks.

Abstract

Sequential structure is a key feature of multiple domains of natural cognition and behavior, such as language, movement and decision-making. Likewise, it is also a central property of tasks to which we would like to apply artificial intelligence. It is therefore of great importance to develop frameworks that allow us to evaluate sequence learning and processing in a domain agnostic fashion, whilst simultaneously providing a link to formal theories of computation and computability. To address this need, we introduce two complementary software tools: SymSeq, designed to rigorously generate and analyze structured symbolic sequences, and SeqBench, a comprehensive benchmark suite of rule-based sequence processing tasks to evaluate the performance of artificial learning systems in cognitively relevant domains. In combination, SymSeqBench offers versatility in investigating sequential structure across diverse knowledge domains, including experimental psycholinguistics, cognitive psychology, behavioral analysis, neuromorphic computing and artificial intelligence. Due to its basis in Formal Language Theory (FLT), SymSeqBench provides researchers in multiple domains with a convenient and practical way to apply the concepts of FLT to conceptualize and standardize their experiments, thus advancing our understanding of cognition and behavior through shared computational frameworks and formalisms. The tool is modular, openly available and accessible to the research community.

Figures (9)

• Figure 1: Conceptual overview of SymSeqBench, the individual components and their interactions.focuses on handling symbolic sequences, specifying processing tasks and enabling the structural analysis of both synthetic and user-provided sequences. SeqWrapper conveniently bundles outputs --- such as input/output sequences and generator objects --- and represents the primary interface for downstream processing. From such wrapper objects, provides functionality for grounding symbolic representations, assigning each symbol to a corresponding discrete- or continuous-time distributed representation, along with additional tasks and analysis that are based on embedded representations. By combining a tool-agnostic design with multiple backends, enables fast and efficient integration with simulation frameworks such as PyTorch and NEST.
• Figure 2: Schematic overview of SymSeq and typical use-case workflows.a) The library consists of four conceptual components that interact through standard Python data structures representing symbolic sequences: The Parser accepts sequential data from the user, either as continuous time-series from which symbolic sequences are built, or as discrete symbolic sequences (string-sets). From first-order transition probabilities, the parser extracts a regular (finite-state) grammar and, resorting to the large battery of analysis metrics, provides extensive and detailed information about the complexity of the sequences and the underlying generative grammar. Language generators comprise various classes for creating task-specific sequences, grouped into synthetic and cognitive/behavioral categories. Both the generated strings and their underlying structure (e.g., transition table) can be directly used by functions in Analysis which provide complexity metrics at multiple scales. The Tasks module is a collection of functions to create target labels for the provided input sequences, organized into recognition and transduction categories; Finally, SeqWrapper serves as the unified input-output interface of , coordinating flow logic and instantiating a complete and ready-to-use experimental setup and dataset. b) SeqWrapper objects can be instantiated from YAML configuration files, allowing subsequent analysis or manipulation of datasets, as well as downstream processing using . c) User-provided sequential data in symbolic form can be analyzed directly or used to infer a regular grammar and create input/output datasets from the corresponding language generator.
• Figure 3: Topological entropy as a measure of grammar complexity.a-d) TE as a function of various grammar properties: a) mean transition density ($|\mathcal{Q}| = 11$); b) grammar size (total number of states in $\mathcal{Q}$, density fixed at $0.25$); c) random grammar with fixed maximum outdegree of one state ($|\mathcal{Q}| = 10$, mean density $0.25$); d) clustering strength ($|\mathcal{Q}| = 100$, $4$ clusters composed of $10$ states, density $0.1$). All results are averaged across 100 trials. e) Convergence of grammar generation with target TE ($0.8$ blue and $2.0$ green, dark shades) and mean transition density ($0.25$ blue and $0.3$ green, light shades). f) Comparison of TE and other standard complexity measures, as a function of the number and depth of ambiguities (transition density $0.25$). Non-structural metrics -- Entropy, Compressibility, Lempel-Ziv-Welch (LZW) and Effective Measure Complexity (EMC) -- were computed on 20000 strings drawn randomly from the respective grammars. Results are averaged across 10 different grammars and normalized individually for each metric. Note that the depth parameter is ignored for no ambiguities.
• Figure 4: Example sequence encodings produced within SeqBench. (a) Discrete token embedding using one-hot vectors. (b) Embedded sequence filtered with an exponential kernel and extended across input channels. (c) Rate-based Poisson spike trains derived from the embedded representation. Encoded continuous signals (b) are considered the mean density (instantaneous firing rate) of a series of independent Poisson processes to translate the signals into spike trains. (d) Google Speech Commands (GSC) representation, where each token is mapped onto one individual spoken command instance. (e) Rate-based spike encoding of GSC. (f) Spiking Speech Commands dataset representation, where each token is mapped to an instance of a spike-coded speech command.
• Figure 5: Schematic overview of SeqBench.provides a modular pipeline for generating, transforming, and analyzing symbolic sequence datasets. A user-defined configuration is passed to the SeqWrapper, which serves as the interface to the sequence generator. The DatasetGenerator either loads existing symbolic data from disk or invokes the SeqWrapper to create new sequences. The SeqDataset forms the core of the system, combining symbolic sequences with base datasets or embeddings, applying perturbations such as gaps and noise, and supporting optional transformation modules for audio, vision, generic tensor operations, or spiking representations. Symbol Mapping defines how tokens correspond to dataset samples or vector embeddings, while the Analysis module computes complexity metrics, including dimensionality and entropy.