ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure

TL;DR

ConPress identifies a reproducible self-compression phenomenon where multi-question prompts induce shorter per-question reasoning traces in large reasoning models. It then leverages this effect with a lightweight self-supervised fine-tuning pipeline that extracts compressed traces under multi-question prompts and distills them into standard single-question inference, without external teachers or reinforcement learning. Empirically, ConPress achieves substantial reductions in reasoning tokens (roughly 30–60%) across diverse math and reasoning benchmarks with minimal accuracy loss, and demonstrates robust behavior across difficulty levels and out-of-distribution data. The work shows that context-induced reasoning efficiency is transferable and can be internalized to enable token-efficient reasoning in practical deployments.

Abstract

Large reasoning models (LRMs) typically solve reasoning-intensive tasks by generating long chain-of-thought (CoT) traces, leading to substantial inference overhead. We identify a reproducible inference-time phenomenon, termed Self-Compression: when multiple independent and answerable questions are presented within a single prompt, the model spontaneously produces shorter reasoning traces for each question. This phenomenon arises from multi-question contextual pressure during generation and consistently manifests across models and benchmarks. Building on this observation, we propose ConPress (Learning from Contextual Pressure), a lightweight self-supervised fine-tuning approach. ConPress constructs multi-question prompts to induce self-compression, samples the resulting model outputs, and parses and filters per-question traces to obtain concise yet correct reasoning trajectories. These trajectories are directly used for supervised fine-tuning, internalizing compressed reasoning behavior in single-question settings without external teachers, manual pruning, or reinforcement learning. With only 8k fine-tuning examples, ConPress reduces reasoning token usage by 59% on MATH500 and 33% on AIME25, while maintaining competitive accuracy.

Figures (8)

• Figure 1: Illustration of single-question and multi-question decoding. By requiring multiple questions to be answered within a single generation, multi-question contexts introduce contextual pressure, which shortens the per-question reasoning traces.
• Figure 2: Distributions of per-question reasoning length under single-question ($N=1$) and two-question ($N=2$) prompting.
• Figure 3: Scaling of self-compression with the number of questions $N$. Top: average reasoning length and relative accuracy. Bottom: reasoning-length distributions across different $N$.
• Figure 4: Effects of ConPress across difficulty levels on MATH500. ConPress consistently reduces reasoning length across all levels while largely preserving accuracy.
• Figure 5: Distribution of reasoning behaviors before and after ConPress training, shown in terms of frequency (left) and normalized density per 100 words (right).