PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries

TL;DR

This work tackles intra-query parallelism in large language models by introducing ParallelPrompt, the first benchmark for uncovering latent parallelism within real user prompts. It presents a scalable data curation and schema extraction pipeline, a three-tier multilingual validation framework, and an evaluation suite that measures latency, semantic fidelity, and speedup under serial versus parallel execution. Across 37k+ prompts spanning canonical and novel categories, the study demonstrates substantial latency reductions (generally three- to five-fold) for many task types, while highlighting limitations such as dependency blindness and language-specific extraction challenges. By releasing the benchmark, curation pipeline, and evaluation tools, the paper provides a standardized testbed to advance structure-aware LLM serving and to explore the integration of semantic parallelism with existing batch- and token-level optimizations in real-world systems.

Abstract

LLM serving systems typically treat user prompts as monolithic inputs, optimizing inference through decoding tricks or inter-query batching. However, many real-world prompts contain latent semantic parallelism--decomposable structures where subtasks can be executed independently to reduce latency while preserving meaning. We introduce PARALLELPROMPT, the first benchmark for measuring intra-query parallelism in natural user prompts. Our dataset comprises over 37,000 real-world prompts from public LLM chat logs, each annotated with a structured schema capturing task templates, shared context, and iteration inputs. These schemas are extracted using LLM-assisted prompting with rule-based multilingual validation. To evaluate the benefits of decomposition, we provide an execution suite that benchmarks serial vs. parallel strategies, measuring latency, structural adherence, and semantic fidelity. Our results show that intra-query parallelism can be successfully parsed in over 75% of curated datasets, unlocking up to 5x speedups on tasks like translation, comprehension, and comparative analysis, with minimal quality degradation. By releasing this benchmark, curation pipeline, and evaluation suite, we provide the first standardized testbed for studying structure-aware execution in LLM serving pipelines.

Figures (10)

• Figure 1: Examples of real user prompts with latent parallel structure. (a) Example of a repeated-generation query, where the 10 generations can be executed in parallel. (b) Example of a classification query, where the task (rating sentences) can be parallelized across the queries (each sentence).
• Figure 2: ParallelPrompt curation pipeline. Our multi-stage filtering process identifies naturally occurring parallelizable structures in real-world LLM interactions. Surprisingly, over 10% of user prompts contain latent parallel structure. High-confidence instances contain explicit structural markers like numbered lists or item delimiters, while medium-confidence rely on semantic cues such as plural forms or task multiplicity. This precision-focused validation ensures benchmark quality for measuring intra-query parallelism benefits.
• Figure 3: Distribution of validated categories by source and validation confidence. The canonical categories dominate (esp. Repeated Generation and Reading Comprehension) since we explicitly optimize for curation of known categories, but the dataset also includes hundreds of emerging, structurally diverse novel categories (e.g., Comparative Analysis, Character Generation). This breadth highlights the benchmark’s coverage of both common and specialized parallelization patterns.
• Figure 4: Validation success rates by category, showing that structured prompts like Named Entity Recognition and Reading Comprehension pass validation more reliably than creative tasks like Repeated Generation. This pattern highlights the limitations of current schema extraction methods for loosely structured or open-ended prompts.
• Figure 5: How ParallelPrompt operationalizes intra-query parallelism. This execution pipeline powers the benchmark's core evaluation, contrasting serial and parallel strategies on real user prompts regardless of their task category. By grounding performance measurement in schema-based decomposition and systematic output comparison, the pipeline reveals when parallel execution achieves meaningful speedups without compromising output quality—turning benchmark abstractions into actionable insights for real-world LLM serving systems.