Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling

TL;DR

Falcon-H1R shows that a 7B parameter model can rival larger SOTA reasoning models when paired with a hybrid Transformer–Mamba architecture and a robust training pipeline combining cold-start SFT and GRPO-based RLVR. The approach leverages long-context fine-tuning, difficulty-aware data curation, and a DeepConf test-time scaling framework to maximize accuracy while reducing inference cost. Across Math, Code, and General reasoning benchmarks, Falcon-H1R-7B achieves top results (e.g., AIME24 88.1%, AIME25 83.1%) with substantially improved token efficiency and fast throughput, outperforming several larger models. This work substantiates the practicality of compact, scalable reasoning backbones for real-world AI systems and provides concrete training and inference optimizations to enable parallel, cost-effective reasoning at scale.

Abstract

This work introduces Falcon-H1R, a 7B-parameter reasoning-optimized model that establishes the feasibility of achieving competitive reasoning performance with small language models (SLMs). Falcon-H1R stands out for its parameter efficiency, consistently matching or outperforming SOTA reasoning models that are $2\times$ to $7\times$ larger across a variety of reasoning-intensive benchmarks. These results underscore the importance of careful data curation and targeted training strategies (via both efficient SFT and RL scaling) in delivering significant performance gains without increasing model size. Furthermore, Falcon-H1R advances the 3D limits of reasoning efficiency by combining faster inference (through its hybrid-parallel architecture design), token efficiency, and higher accuracy. This unique blend makes Falcon-H1R-7B a practical backbone for scaling advanced reasoning systems, particularly in scenarios requiring extensive chain-of-thoughts generation and parallel test-time scaling. Leveraging the recently introduced DeepConf approach, Falcon-H1R achieves state-of-the-art test-time scaling efficiency, offering substantial improvements in both accuracy and computational cost. As a result, Falcon-H1R demonstrates that compact models, through targeted model training and architectural choices, can deliver robust and scalable reasoning performance.

Figures (9)

• Figure 1: DeepConf@512 deepconf average results over AIME24, AIME25, AMO-Bench, and GPQA-Diamond (Detailed results in Table \ref{['tab:bench']}). https://huggingface.co/tiiuae/Falcon-H1R-7B achieves exceptional performance by pushing the reasoning frontiers in 3 dimensions: higher accuracy, token efficiency and fast inference in the parallel thinking setting.
• Figure 2: Distribution of response token counts per domain in the SFT stage. The "Other" category includes IF, Chat, Safety, and Tool calling data.
• Figure 3: Distribution of data categories in the SFT stage. The "Other" category includes IF, Chat, Safety, and Tool calling data.
• Figure 4: Effect of enabling Data-Parallel Balance on downstream reasoning task.
• Figure 5: Math prompts difficulty distribution relative to the Falcon-H1R-SFT checkpoint.