Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Batch-of-Thought: Cross-Instance Learning for Enhanced LLM Reasoning

Xuan Yang, Furong Jia, Roy Xie, Xiong Xi, Hengwei Bian, Jian Li, Monica Agrawal

TL;DR

Batch-of-Thought (BoT) is a training-free framework that enables cross-instance learning by processing related queries as batches, enabling comparative reasoning, pattern transfer, and better uncertainty calibration. BoT-R instantiates this in a two-agent Actor–Reflector setup where the Actor generates initial reasoning and the Reflector jointly evaluates the batch to refine or finalize answers, leading to improved accuracy and confidence, plus substantial cost reductions. The approach yields consistent gains across three model families and six benchmarks, with average token-cost reductions up to 61% and improved calibration as evidenced by KS and ECE metrics. The work provides theoretical foundations showing information gains and Pareto improvements from batch-aware reasoning and demonstrates practical applicability, including a new Seller Fraud Detection benchmark for high-stakes agentic reasoning; limitations include batch coherence requirements and context-window constraints.

Abstract

Current Large Language Model reasoning systems process queries independently, discarding valuable cross-instance signals such as shared reasoning patterns and consistency constraints. We introduce Batch-of-Thought (BoT), a training-free method that processes related queries jointly to enable cross-instance learning. By performing comparative analysis across batches, BoT identifies high-quality reasoning templates, detects errors through consistency checks, and amortizes computational costs. We instantiate BoT within a multi-agent reflection architecture (BoT-R), where a Reflector performs joint evaluation to unlock mutual information gain unavailable in isolated processing. Experiments across three model families and six benchmarks demonstrate that BoT-R consistently improves accuracy and confidence calibration while reducing inference costs by up to 61%. Our theoretical and experimental analysis reveals when and why batch-aware reasoning benefits LLM systems.

Batch-of-Thought: Cross-Instance Learning for Enhanced LLM Reasoning

TL;DR

Batch-of-Thought (BoT) is a training-free framework that enables cross-instance learning by processing related queries as batches, enabling comparative reasoning, pattern transfer, and better uncertainty calibration. BoT-R instantiates this in a two-agent Actor–Reflector setup where the Actor generates initial reasoning and the Reflector jointly evaluates the batch to refine or finalize answers, leading to improved accuracy and confidence, plus substantial cost reductions. The approach yields consistent gains across three model families and six benchmarks, with average token-cost reductions up to 61% and improved calibration as evidenced by KS and ECE metrics. The work provides theoretical foundations showing information gains and Pareto improvements from batch-aware reasoning and demonstrates practical applicability, including a new Seller Fraud Detection benchmark for high-stakes agentic reasoning; limitations include batch coherence requirements and context-window constraints.

Abstract

Current Large Language Model reasoning systems process queries independently, discarding valuable cross-instance signals such as shared reasoning patterns and consistency constraints. We introduce Batch-of-Thought (BoT), a training-free method that processes related queries jointly to enable cross-instance learning. By performing comparative analysis across batches, BoT identifies high-quality reasoning templates, detects errors through consistency checks, and amortizes computational costs. We instantiate BoT within a multi-agent reflection architecture (BoT-R), where a Reflector performs joint evaluation to unlock mutual information gain unavailable in isolated processing. Experiments across three model families and six benchmarks demonstrate that BoT-R consistently improves accuracy and confidence calibration while reducing inference costs by up to 61%. Our theoretical and experimental analysis reveals when and why batch-aware reasoning benefits LLM systems.
Paper Structure (53 sections, 5 theorems, 16 equations, 1 figure, 8 tables, 1 algorithm)

This paper contains 53 sections, 5 theorems, 16 equations, 1 figure, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Problem Formulation
  4. Actor $\mathcal{A}$.
  5. Reflector $\mathcal{R}$.
  6. Objective.
  7. Batch-of-Thought (BoT)
  8. Formalization.
  9. Cross-instance mechanisms.
  10. Theoretical foundation.
  11. Experiments
  12. Experimental Setup
  13. Datasets.
  14. Metrics.
  15. Baselines.
  16. ...and 38 more sections

Key Result

Theorem C.3

Let $\mathcal{G}_0 = \sigma(\hat{y}_i^{(0)}, \rho_i)$ represent the $\sigma$-algebra generated by instance $i$ alone, and $\mathcal{G}_1 = \sigma(\hat{y}_i^{(0)}, \rho_i, \phi)$ where $\phi = \phi(\mathcal{C}^{\mathrm{BoT}})$ denotes batch-level statistics. For any strictly proper scoring rule $S: [ satisfies where $u^{\mathrm{ind}} = \mathbb{P}(\hat{y}_i^{(0)} = y_i^* \mid \mathcal{G}_0)$ and $z

Figures (1)

  • Figure 1: Batch-of-Thought reflection framework. An Actor generates initial responses for a batch of queries. A Reflector then jointly evaluates all responses through comparative analysis, determining whether each should be finalized or refined with feedbacks.

Theorems & Definitions (11)

  • Definition C.1: Batch reasoning problem
  • Theorem C.3: Batch processing improves proper scoring rules
  • proof
  • Remark C.4: Connection to Expected Calibration Error
  • Proposition C.5: Effective sample size under correlation
  • proof
  • Corollary C.6: Confidence separation for error detection
  • Remark C.7: Optimal batch composition
  • Proposition C.8: Sublinear cost scaling
  • Theorem C.9: Conditions for BoT effectiveness
  • ...and 1 more