StreamingThinker: Large Language Models Can Think While Reading

TL;DR

The paper tackles the latency and attention-inefficiency of batch reasoning in large language models by introducing streaming thinking, a paradigm that enables reasoning to unfold concurrently with input as it arrives. It presents StreamingThinker, a framework integrating streaming CoT generation, streaming-aware training, and streaming inference with parallel KV caches and streaming-specific masks/encoding. Across math, logical, and long-context QA tasks on Qwen3 models, StreamingThinker achieves performance comparable to batch thinking while dramatically reducing token waiting (about 80%) and overall latency (over 60%), demonstrating the practicality of real-time, streaming reasoning. The work also provides detailed design elements (boundary tokens, depth-controlled reasoning, and parallel inference) and extensive empirical analyses, highlighting the potential of streaming thinking to bridge human-like cognition and scalable AI reasoning in dynamic settings.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in chain of thought (CoT) reasoning. However, the current LLM reasoning paradigm initiates thinking only after the entire input is available, which introduces unnecessary latency and weakens attention to earlier information in dynamic scenarios. Inspired by human cognition of thinking while reading, we first design a \textit{\textbf{streaming thinking}} paradigm for LLMs, where reasoning unfolds in the order of input and further adjusts its depth once reading is complete. We instantiate this paradigm with \textit{StreamingThinker}, a framework that enables LLMs to think while reading through the integration of streaming CoT generation, streaming-constraint training, and streaming parallel inference. Specifically, StreamingThinker employs streaming reasoning units with quality control for CoT generation, enforces order-preserving reasoning through streaming attention masks and position encoding, and leverages parallel KV caches that decouple input encoding from reasoning generation, thereby ensuring alignment and enabling true concurrency. We evaluate StreamingThinker on the Qwen3 model family across math reasoning, logical reasoning, and context-based QA reasoning tasks. Experimental results show that the StreamingThinker preserves performance comparable to batch thinking, while yielding an 80\% reduction in token waiting before the onset of reasoning and a more than 60\% reduction in time-level latency for producing the final answer, demonstrating the effectiveness of the streaming paradigm for LLM reasoning. Code will be released at https://github.com/EIT-NLP/StreamingLLM/tree/main/StreamingThinker.

Figures (7)

• Figure 1: (a) Standard LLM reasoning follows the batch thinking paradigm, where reasoning begins only after the entire input is received, leading to high latency and imbalanced attention to the input. The proposed streaming thinking paradigm enables LLMs to think while reading during input reception, substantially reducing latency and maintaining attention aligned with the order of input. (b) Streaming thinking paradigm supports multi-depth reasoning, balancing latency with performance.
• Figure 1: Examples of sentence-level similarity map between the input sentences and the reasoning content (from math reasoning task). High similarity along the diagonal demonstrates strong alignment, enabling evaluation through sequential consistency.
• Figure 2: Generation for streaming CoT.
• Figure 2: Comparison of reasoning paradigms. Streaming thinking enables concurrent reading and reasoning with minimal delay, while interleaved thinking alternates between the two but still accumulates overhead, and batch thinking postpones reasoning until all inputs are read, leading to the largest delay.
• Figure 3: Training and inference framework of StreamingThinker. (a) shows attention mask at training. (b) and (c) show attention routing and parallel KV caches for streaming thinking at inference.