Towards a Systems Theory of Algorithms

TL;DR

The traditional view treats numerical algorithms as isolated code; this paper proposes a systems-theoretic perspective in which algorithms are open dynamical systems with inputs $u$, outputs $y$, state $x$, and disturbances $\eta$ that interact with real-time environments. It develops a vision for a systems theory of algorithms, surveys historical and contemporary case studies across optimization, learning, real-time control, and decision-making, and outlines key modeling, interconnection, architecture, and synthesis challenges. By applying tools such as dissipativity and small-gain theory to open, interconnected algorithms, the work demonstrates how stability, robustness, and performance can be analyzed in online, networked, and cyber-physical contexts, linking algorithm design to control-stack concepts. The proposed viewpoint seeks to bridge control theory and algorithmic design, enabling principled, scalable, and collaborative advances for complex in vivo computational systems.

Abstract

Traditionally, numerical algorithms are seen as isolated pieces of code confined to an {\em in silico} existence. However, this perspective is not appropriate for many modern computational approaches in control, learning, or optimization, wherein {\em in vivo} algorithms interact with their environment. Examples of such {\em open algorithms} include various real-time optimization-based control strategies, reinforcement learning, decision-making architectures, online optimization, and many more. Further, even {\em closed} algorithms in learning or optimization are increasingly abstracted in block diagrams with interacting dynamic modules and pipelines. In this opinion paper, we state our vision on a to-be-cultivated {\em systems theory of algorithms} and argue in favor of viewing algorithms as open dynamical systems interacting with other algorithms, physical systems, humans, or databases. Remarkably, the manifold tools developed under the umbrella of systems theory are well suited for addressing a range of challenges in the algorithmic domain. We survey various instances where the principles of algorithmic systems theory are being developed and outline pertinent modeling, analysis, and design challenges.

Figures (10)

• Figure 1: We advocate modelling an algorithm as an open discrete-time dynamical system subject to inputs $u$, outputs $y$, an internal latent variable (state) $x$, and exogeneous signal $\eta$ collecting different sources of uncertainty.
• Figure 2: The Kalman filtering approach \ref{['eq: Kalman filter']} is an online algorithm to solve the least-squares problem \ref{['eq:LS']} with streaming data.
• Figure 3: The block diagram illustrates the saddle-point dynamics \ref{['eq:saddle_point_dynamics']}.
• Figure 4: The block diagram illustrates the structure \ref{['eq:gd_alg_family']} of gradient-based optimization algorithms, where $I_n$ denotes the identity matrix of size $n$.
• Figure 5: The block diagram adapted from alonso2024state shows the structured state-space sequence model. Stacks of such models form a foundation model.

Theorems & Definitions (7)

• Example 1: Primal-dual algorithms as proportional-integral controllers
• Example 2: Gradient-based algorithms as closed-loop systems
• Example 3: Feedforward for time-varying optimization
• Example 4: Structured state-space sequence model
• Example 5: Sub-optimal Model Predictive Control
• Example 6: Distributed Optimization via Gradient Tracking
• Example 7: Online Feedback Optimization