Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EdgeRunner 20B: Military Task Parity with GPT-5 while Running on the Edge

Jack FitzGerald, Aristotelis Lazaridis, Dylan Bates, Aman Sharma, Jonnathan Castillo, Yousif Azami, Sean Bailey, Jeremy Cao, Peter Damianov, Kevin de Haan, Luke Kerbs, Vincent Lu, Joseph Madigan, Jeremy McLaurin, Jonathan Tainer, Dave Anderson, Jonathan Beck, Jamie Cuticello, Colton Malkerson, Tyler Saltsman

TL;DR

EdgeRunner 20B demonstrates that a tightly specialized, edge-deployed LLM can achieve military-task parity with GPT-5 while running on air-gapped hardware, by training on a curated $1.6\times 10^{6}$-record military corpus. The authors introduce four military-specific test sets alongside broad general-purpose benchmarks, and employ a multi-stage data-generation pipeline together with the Inspect evaluation framework to quantify performance and reliability. Key findings show non-regression on most general benchmarks and competitive performance on military tasks, with a few exceptions, as well as compelling insights into hyperparameters, cost, and throughput that favor edge deployments for data-sensitive operations. The work also outlines a roadmap for expanded evaluation, smaller models, and continued security-focused development to support resilient, on-device military AI systems.

Abstract

We present EdgeRunner 20B, a fine-tuned version of gpt-oss-20b optimized for military tasks. EdgeRunner 20B was trained on 1.6M high-quality records curated from military documentation and websites. We also present four new tests sets: (a) combat arms, (b) combat medic, (c) cyber operations, and (d) mil-bench-5k (general military knowledge). On these military test sets, EdgeRunner 20B matches or exceeds GPT-5 task performance with 95%+ statistical significance, except for the high reasoning setting on the combat medic test set and the low reasoning setting on the mil-bench-5k test set. Versus gpt-oss-20b, there is no statistically-significant regression on general-purpose benchmarks like ARC-C, GPQA Diamond, GSM8k, IFEval, MMLU Pro, or TruthfulQA, except for GSM8k in the low reasoning setting. We also present analyses on hyperparameter settings, cost, and throughput. These findings show that small, locally-hosted models are ideal solutions for data-sensitive operations such as in the military domain, allowing for deployment in air-gapped edge devices.

EdgeRunner 20B: Military Task Parity with GPT-5 while Running on the Edge

TL;DR

EdgeRunner 20B demonstrates that a tightly specialized, edge-deployed LLM can achieve military-task parity with GPT-5 while running on air-gapped hardware, by training on a curated -record military corpus. The authors introduce four military-specific test sets alongside broad general-purpose benchmarks, and employ a multi-stage data-generation pipeline together with the Inspect evaluation framework to quantify performance and reliability. Key findings show non-regression on most general benchmarks and competitive performance on military tasks, with a few exceptions, as well as compelling insights into hyperparameters, cost, and throughput that favor edge deployments for data-sensitive operations. The work also outlines a roadmap for expanded evaluation, smaller models, and continued security-focused development to support resilient, on-device military AI systems.

Abstract

We present EdgeRunner 20B, a fine-tuned version of gpt-oss-20b optimized for military tasks. EdgeRunner 20B was trained on 1.6M high-quality records curated from military documentation and websites. We also present four new tests sets: (a) combat arms, (b) combat medic, (c) cyber operations, and (d) mil-bench-5k (general military knowledge). On these military test sets, EdgeRunner 20B matches or exceeds GPT-5 task performance with 95%+ statistical significance, except for the high reasoning setting on the combat medic test set and the low reasoning setting on the mil-bench-5k test set. Versus gpt-oss-20b, there is no statistically-significant regression on general-purpose benchmarks like ARC-C, GPQA Diamond, GSM8k, IFEval, MMLU Pro, or TruthfulQA, except for GSM8k in the low reasoning setting. We also present analyses on hyperparameter settings, cost, and throughput. These findings show that small, locally-hosted models are ideal solutions for data-sensitive operations such as in the military domain, allowing for deployment in air-gapped edge devices.

Paper Structure

This paper contains 19 sections, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Test Sets
  3. Military-Specific Test Sets
  4. General-Purpose Test Sets
  5. Methods
  6. Training Data
  7. Training Setup
  8. Evaluation Setup
  9. Results and Analysis
  10. Fine Tuning Accuracy
  11. General Purpose Tasks
  12. Hyperparameter Analysis
  13. Cloud Cost Analysis
  14. Throughput Analysis
  15. Related Work
  16. ...and 4 more sections

Figures (6)

  • Figure 1: The EdgeRunner inference system for running military-specific LLMs locally on the user's device and performing Retrieval Augmented Generation (RAG).
  • Figure 2: A visual representation of the data presented in Table \ref{['tab:mil-results']} showing the relative error of EdgeRunner's fine tuned gpt-oss-20b model relative to GPT-5 with medium reasoning. Error bars are given for the standard error. Lower numbers are better (less error).
  • Figure 3: A visual representation of the data presented in Table \ref{['tab:gen-results']} showing the relative error of EdgeRunner's fine tuned gpt-oss-20b model relative to GPT-5 with medium reasoning. Error bars are given for the standard error. Lower numbers are better (less error).
  • Figure 4: The alpaca chat template.
  • Figure 5: An approximated version of the task definition used for all four military test sets.
  • ...and 1 more figures