Chimera: State Space Models Beyond Sequences

TL;DR

Chimera presents a principled framework that extends State Space Models to directly encode data topology by leveraging the resolvent of a graph's adjacency matrix, thereby unifying modeling across language, vision, and graphs without domain-specific biases. The core idea fuses SMA-based recurrences with graph topology through $\mathbf{L}=(\mathbf{I}-\mathbf{A})^{-1}$, enabling outputs $\mathbf{y} = (\mathbf{L} \odot (\mathbf{C}\bar{\mathbf{B}}^T))\mathbf{V}$ that aggregate influence along all paths. The paper introduces efficient implementations for DAGs (linear-time recurrence and matrix-based acceleration) and a finite-sum Neumann-series approximation for general graphs, preserving performance while controlling cost. Empirically, Chimera achieves strong results on MLM, ImageNet-1k, and Long Range Graph Benchmark, often outperforming transformer baselines and demonstrating the value of exploiting topology as an inductive bias. The work lays a foundation for topology-aware modeling with practical extensions, while noting cubic-cost challenges for general graphs and suggesting hardware-aware optimizations and broader domain applications as future directions.

Abstract

Transformer-based deep learning methods have become the standard approach for modeling diverse data such as sequences, images, and graphs. These methods rely on self-attention, which treats data as an unordered set of elements. This ignores the neighborhood structure or graph topology of the data and requires inductive biases--such as position embeddings in sequences and images, or random walks in graphs--to incorporate topology. However, designing such task-specific biases requires significant effort and can introduce side effects that hinder generalization. We introduce Chimera, a unified model that directly incorporates data topology in a principled way, removing the need for domain-specific biases. The key idea is that state space models--which naturally do not require position embeddings--can be generalized to capture any graph topology. Our experiments show that Chimera achieves strong performance across language, vision, and graph domains, outperforming BERT on GLUE by 0.7 points, ViT on ImageNet-1k by 2.6%, and all baselines on the Long Range Graph Benchmark. We further propose algorithmic optimizations to improve Chimera's efficiency: (1) for Directed Acyclic Graphs, Chimera can be implemented as a linear-time recurrence; (2) for general graphs, a simple mathematical relaxation achieves Transformer's quadratic complexity without domain-specific heuristics. These results validate Chimera's core contribution and support the idea that data topology is a powerful inductive bias across modalities.

Figures (6)

• Figure 1: SSMs inherently operate on a directed line graph topology: SSMs modeling a sequence of tokens in the recurrent representation (left), the structured mask matrix from the SMA representation of SSMs (center), The underlying directed line graph topology (right).
• Figure 2: Canonical DAG decomposition of undirected line graph topology (left) into two directed line graph topologies (right).
• Figure 3: Grid graph (left). The canonical 2D-DAG decomposition of the grid graph (right). These graphs are sufficient to capture the influence between all pairs of nodes in the undirected grid graph.
• Figure 4: Chimera on DAGs: A visualization of the normalized recurrence. The absence of cycles in DAGs enables a recurrent view of the method which allows for a fast linear-time computation.
• Figure 5: Progressively destroying the 2D grid graph topology. Fwd & Rev (top): 1D flattened grid with bidirectional edges. Fwd (bottom): 1D flattened grid graph with only forward edges.

Theorems & Definitions (15)

• Definition 3.1: Resolvent of a Linear Operator reed1980methods
• Proposition 3.2: ${\bm{A}}^k$ accumulate influence through paths of length $k$
• Proposition 3.3
• Proposition 3.4
• Proposition 4.1
• Proposition 4.2
• Proposition 4.3
• Proposition 4.4
• Theorem 4.5
• Proposition 4.6