EdgeJury: Cross-Reviewed Small-Model Ensembles for Truthful Question Answering on Serverless Edge Inference

TL;DR

EdgeJury tackles the persistent problem of hallucinations in QA by enabling a four-stage, edge-friendly ensemble of small language models. It couples role-specific parallel generation, anonymized cross-review, a chairman synthesis step, and a claim-level consistency verifier to produce truthful answers without external retrieval. The approach yields substantial improvements on TruthfulQA MC1 and adversarial EdgeCases, with manual analyses showing reduced factual errors and high reliability signals, all demonstrated on Cloudflare Workers AI. The work highlights the practicality and value of structured multi-model interaction at the edge, offering a scalable alternative to large-model or retrieval-heavy pipelines for truth-sensitive QA.

Abstract

Hallucinations hinder reliable question answering, especially in resource-constrained deployments where frontier-scale models or retrieval pipelines may be impractical. We present EdgeJury, a lightweight ensemble framework that improves truthfulness and robustness using only small instruction-tuned language models (3B-8B) suitable for serverless edge inference. EdgeJury orchestrates four stages: (1) parallel role-specialized generation, (2) anonymized cross-review with structured critiques and rankings, (3) chairman synthesis that integrates the strongest content while addressing flagged issues, and (4) claim-level consistency labeling based on inter-model agreement. On TruthfulQA (MC1), EdgeJury achieves 76.2% accuracy (95% CI: 72.8-79.6%), a +21.4% relative improvement over a single 8B baseline (62.8%), and outperforms standard baselines including self-consistency and majority voting under transparent compute accounting (total tokens and platform cost reported). On a 200-question adversarial EdgeCases set, EdgeJury yields +48.2% relative gains (95% CI: 44.0-52.4%). Manual analysis on 100 incorrect answers shows an approximately 55% reduction in factual hallucination errors versus the single-model baseline. Deployed on Cloudflare Workers AI, EdgeJury achieves 8.4 s median end-to-end latency, demonstrating that coordinated small-model ensembles can improve truthfulness on misconception-heavy QA benchmarks without external retrieval or proprietary large-model APIs.

Figures (3)

• Figure 1: EdgeJury four-stage ensemble pipeline. (a) Stage 1: Multiple small LLMs (3B--8B) generate role-specialized answers in parallel. (b) Stage 2: Models anonymously cross-review peers, producing structured rankings and critiques. (c) Stage 3: A chairman model synthesizes a final answer using candidate responses and review feedback. (d) Stage 4: An agreement-based verifier extracts atomic claims and labels each as consistent, uncertain, or contradicted based on inter-model agreement.
• Figure 2: Deployment architecture on Cloudflare Workers AI. A client sends the query to a nearby Worker instance. The Worker orchestrates Stages 1--4 by invoking Workers AI model endpoints. Optional logging (e.g., request traces) can be integrated via a lightweight database, but is not required for inference. Arrows indicate data flows: Query $\rightarrow$ Worker $\rightarrow$ Model Endpoints.
• Figure 3: Accuracy comparison. EdgeJury outperforms baselines, with the largest gains on adversarial EdgeCases.