Towards a Multimodal Stream Processing System

TL;DR

This work addresses the challenge of enabling real-time multimodal analytics in streaming systems by introducing Samsāra, a super-optimizer that coalesces semantic, logical, and physical optimizations to run multimodal queries efficiently. By leveraging MLLMs as first-class operators and automating data-reduction strategies through LLM-guided reasoning, the approach achieves substantial throughput gains while controlling accuracy degradation. The Toll Booth case study and end-to-end results on 13 queries demonstrate the potential of deep plan specialization for streaming workloads. The paper also outlines a roadmap for formalizing semantic rewrites, adaptive model synthesis, and broader multimodal streaming capabilities, highlighting both the opportunities and research challenges in this nascent area.

Abstract

In this paper, we present a vision for a new generation of multimodal streaming systems that embed MLLMs as first-class operators, enabling real-time query processing across multiple modalities. Achieving this is non-trivial: while recent work has integrated MLLMs into databases for multimodal queries, streaming systems require fundamentally different approaches due to their strict latency and throughput requirements. Our approach proposes novel optimizations at all levels, including logical, physical, and semantic query transformations that reduce model load to improve throughput while preserving accuracy. We demonstrate this with Samsara, a prototype leveraging such optimizations to improve performance by more than an order of magnitude. Moreover, we discuss a research roadmap that outlines open research challenges for building a scalable and efficient multimodal stream processing systems.

Figures (5)

• Figure 1: Illustrating multimodal plan optimizations proposed by Samsāra in (a) with evaluation results in (b). Naively integrating MLLMs into stream processing leads to extremely low performance. With Samsāra and its novel optimizations for multimodal data streams, we achieve up to $9\times$ performance improvements for this example.
• Figure 2: Overview of Samsāra, a super-optimizer to enable multimodal streaming systems which transforms a naive multimodal query plan into an optimized plan through a series of novel semantic, logical, and physical optimizations which aggressively optimize the plan for a particular long-running streaming query plan.
• Figure 3: Overview of the semantic optimization procedure applied to the tollbooth query. Starting from a naive plan (Extract + Filter), the optimizer leverages an LLM to perform three reasoning phases: (1) World-Knowledge Extraction, where it identifies relevant entities (cars), their properties (color, size, speed), and real-world dynamics (limited movement through the tollbooth); (2) Operator Selection, where it proposes semantically motivated operators such as Crop, Downscale, and Skip based on extracted knowledge and metadata; and (3) Plan Update, where the chosen operators are iteratively inserted to yield a semantically optimized plan (Crop + Extract + Filter). The figure shows how LLM-guided reasoning bridges data semantics and query intent to automatically produce more efficient execution plans.
• Figure 4: Overview of logical optimiseation in Samsāra. It starts with a semantically optimized plan with Skip, Scale, Crop, MLLM, and Filter operators to optimize it analogously to the relational counterparts, i.e., (1) Data model reconciliation that maps images to tuples, (2) Operator mapping that maps multimodal operators to relational and (3) Optimization rule mapping that applies logical rewrite rules. The resulting logically optimiseed plan is highlighted on the right.
• Figure 5: End-to-End gains for all(13) queries cf Table \ref{['exp:queries']}. Samsāra has up to $10\times$ speedup over naive execution