JAF: Judge Agent Forest

TL;DR

Judge Agent Forest (JAF) reframes the judge in agentic AI from an instance-local evaluator to a cohort-level reasoner that jointly evaluates related query–response pairs. It leverages overlapping neighborhoods induced by a learned locality-sensitive hashing (LSH) scheme to form a cohort-level knowledge graph, enabling cross-instance critique propagation and ensemble-like robustness across multiple passes. The approach combines information-theoretic hash predicates, divergence-based and OOD-aware splits, and CoT sharing to improve accuracy and calibration in complex, context-rich triage tasks, demonstrated on cloud misconfiguration triage. While compatible with reinforcement learning and supervised fine-tuning, JAF primarily operates at inference time to denoise and align outputs across a workload, offering scalable, interpretable, and privacy-friendly cohort-aware refinement with measurable improvements over isolated judge baselines.

Abstract

Judge agents are fundamental to agentic AI frameworks: they provide automated evaluation, and enable iterative self-refinement of reasoning processes. We introduce JAF: Judge Agent Forest, a framework in which the judge agent conducts joint inference across a cohort of query--response pairs generated by a primary agent, rather than evaluating each in isolation. This paradigm elevates the judge from a local evaluator to a holistic learner: by simultaneously assessing related responses, the judge discerns cross-instance patterns and inconsistencies, whose aggregate feedback enables the primary agent to improve by viewing its own outputs through the judge's collective perspective. Conceptually, JAF bridges belief propagation and ensemble-learning principles: overlapping in-context neighborhoods induce a knowledge-graph structure that facilitates propagation of critique, and repeated, randomized evaluations yield a robust ensemble of context-sensitive judgments. JAF can be instantiated entirely via ICL, with the judge prompted for each query using its associated primary-agent response plus a small, possibly noisy set of peer exemplars. While kNN in embedding space is a natural starting point for exemplars, this approach overlooks categorical structure, domain metadata, or nuanced distinctions accessible to modern LLMs. To overcome these limitations, we develop a flexible locality-sensitive hashing (LSH) algorithm that learns informative binary codes by integrating semantic embeddings, LLM-driven hash predicates, supervision from categorical labels, and relevant side information. These hash codes support efficient, interpretable, and relation-aware selection of diverse exemplars, and further optimize exploration of CoT reasoning paths. We validate JAF with an empirical study on the demanding task of cloud misconfigs triage in large-scale cloud environments.

Figures (9)

• Figure 1: High-level contrast between standard instance-local use of a judge agent and the joint, cohort-level perspective adopted in Judge Agent Forest (JAF). In panel (a), each query $Q_i$ and primary-agent response $R_i$ is evaluated by a judge in isolation; information does not flow across instances. In panel (b), a single judge agent reviews each $(Q_i,R_i)$ while also seeing a small set of other query--response pairs from the same tenant- or account-scoped cohort in its prompt. This richer context allows the judge to compare decisions across related issues, detect inconsistencies, and propagate useful critique patterns, without changing the underlying LLM architecture.
• Figure 2: Conceptual overview of JAF operating on a cohort of queries. Nodes $Q_1,\dots,Q_8$ (top row) represent queries (e.g., vulnerability--asset pairs), and nodes $R_1,\dots,R_6$ (middle row) represent the corresponding primary-agent responses. Each judge invocation $J^1,J^2,J^3,J^4$ (bottom row) reviews one focal response while also receiving a random, relation-aware subset of other query--response pairs, indicated by the dashed connections. Across the cohort, these overlapping neighborhoods induce a "forest" of judge contexts: each $J^k$ is a tree rooted at a focal instance, with branches to its in-context exemplars. Repeated passes over the same cohort (or over successive cohorts from the same tenant or account) (or multiple evaluation runs) allow information to propagate along these connections, leading to more calibrated and consistent judgments than isolated, one-shot reviews.
• Figure 3: How Judge Agent Forest (JAF) combines iterative self-refinement and exploration of multiple chains of thought. Panel (a) illustrates a conventional self-critique loop for a single query $Q_1$, where a judge reviews a primary-agent response, produces feedback, and the primary agent updates its answer madaan2023selfshinn2023reflexion. Panel (b) depicts standard CoT exploration, where several independent reasoning paths ($r^1,r^2,\dots$) are generated and later aggregated wei2022cotwang2022selfconsistency. Panel (c) shows JAF's perspective: the judge sees multiple CoT variants and multiple related queries simultaneously, enabling it to compare reasoning patterns across both dimensions. This joint view allows the judge to highlight inconsistencies, reuse successful reasoning templates, and guide refinement in a way that is sensitive to the overall distribution of problem instances within a tenant or account cohort, not just a single case.
• Figure 4: A sketch of locality-sensitive hashing (LSH) applied to joint representations of query--response pairs indyk1998approximateandoni2015practicalgarg2019kernelizedgarg2020modeling. Each point corresponds to a pair $(Q_i,R_i)$ embedded in a semantic space that reflects both the input (e.g., vulnerability context, asset metadata) and salient features of the primary-agent response (e.g., cited evidence, inferred exposure). Dashed lines represent randomized hash hyperplanes; together they define binary hash codes (buckets) such that nearby points are likely to share the same code. Within JAF, these buckets are used to efficiently select informative and diverse in-context exemplars for the judge: for a focal pair $(Q_i,R_i)$, we sample other pairs from its own and neighboring buckets to populate the judge's prompt. This organizes the cohort into a structure that is both computationally tractable and aligned with the underlying semantics and domain structure of the task.
• Figure 5: Conceptual view of how LSH can be used to guide exploration of diverse chains of thought (CoTs) within JAF openreview2025scalingtestarxiv2025artscalingaclanthology2025s1garg2023clustering. Each node in the tree represents a partial or complete reasoning path for a query; these paths are embedded and hashed into binary codes (e.g., 000, 001, …, 111) according to their semantic and procedural characteristics. Buckets with many CoTs correspond to frequently explored reasoning styles, while sparsely populated buckets capture under-explored or atypical strategies. By monitoring which hash codes are under-sampled and how they correlate with judged quality, JAF can allocate additional test-time compute to promising yet less-visited regions of the reasoning space, optionally using reinforcement learning arxiv2024explorationstanford2018scalable to learn exploration policies. This allows the system to balance exploitation of known-good reasoning templates with exploration of novel ones, all within a principled, LSH-organized framework that is aligned with the emerging knowledge graph over queries and CoTs.