Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment

Charlie Masters, Marta Grześkiewicz, Stefano V. Albrecht

TL;DR

ARCANE reframes alignment as a multi-agent, rubric-based problem where a manager elicits and learns structured, interpretable rubrics that guide worker policies at test time. Through a two-stage curriculum—supervised fine-tuning and Group Sequence Policy Optimization—the framework produces compact, verifiable criteria that align outputs with stakeholder utilities without retraining the workers. Empirical results on GDPVal show that rubric-guided policies improve usefulness and faithfulness of rankings, while maintaining interpretability and auditability, with performance approaching an oracle rubric. The work highlights the practical value of dynamic, human-interpretable objectives for trustworthy, long-horizon AI systems and discusses future work on scaling and multi-agent coordination.

Abstract

As agents based on large language models are increasingly deployed to long-horizon tasks, maintaining their alignment with stakeholder preferences becomes critical. Effective alignment in such settings requires reward models that are interpretable so that stakeholders can understand and audit model objectives. Moreover, reward models must be capable of steering agents at interaction time, allowing preference shifts to be incorporated without retraining. We introduce ARCANE, a framework that frames alignment as a multi-agent collaboration problem that dynamically represents stakeholder preferences as natural-language rubrics: weighted sets of verifiable criteria that can be generated on-the-fly from task context. Inspired by utility theory, we formulate rubric learning as a reconstruction problem and apply a regularized Group-Sequence Policy Optimization (GSPO) procedure that balances interpretability, faithfulness, and computational efficiency. Using a corpus of 219 labeled rubrics derived from the GDPVal benchmark, we evaluate ARCANE on challenging tasks requiring multi-step reasoning and tool use. The learned rubrics produce compact, legible evaluations and enable configurable trade-offs (e.g., correctness vs. conciseness) without retraining. Our results show that rubric-based reward models offer a promising path toward interpretable, test-time adaptive alignment for complex, long-horizon AI systems.

ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment

TL;DR

ARCANE reframes alignment as a multi-agent, rubric-based problem where a manager elicits and learns structured, interpretable rubrics that guide worker policies at test time. Through a two-stage curriculum—supervised fine-tuning and Group Sequence Policy Optimization—the framework produces compact, verifiable criteria that align outputs with stakeholder utilities without retraining the workers. Empirical results on GDPVal show that rubric-guided policies improve usefulness and faithfulness of rankings, while maintaining interpretability and auditability, with performance approaching an oracle rubric. The work highlights the practical value of dynamic, human-interpretable objectives for trustworthy, long-horizon AI systems and discusses future work on scaling and multi-agent coordination.

Abstract

As agents based on large language models are increasingly deployed to long-horizon tasks, maintaining their alignment with stakeholder preferences becomes critical. Effective alignment in such settings requires reward models that are interpretable so that stakeholders can understand and audit model objectives. Moreover, reward models must be capable of steering agents at interaction time, allowing preference shifts to be incorporated without retraining. We introduce ARCANE, a framework that frames alignment as a multi-agent collaboration problem that dynamically represents stakeholder preferences as natural-language rubrics: weighted sets of verifiable criteria that can be generated on-the-fly from task context. Inspired by utility theory, we formulate rubric learning as a reconstruction problem and apply a regularized Group-Sequence Policy Optimization (GSPO) procedure that balances interpretability, faithfulness, and computational efficiency. Using a corpus of 219 labeled rubrics derived from the GDPVal benchmark, we evaluate ARCANE on challenging tasks requiring multi-step reasoning and tool use. The learned rubrics produce compact, legible evaluations and enable configurable trade-offs (e.g., correctness vs. conciseness) without retraining. Our results show that rubric-based reward models offer a promising path toward interpretable, test-time adaptive alignment for complex, long-horizon AI systems.

Paper Structure

This paper contains 48 sections, 18 equations, 1 figure, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Training-Time Alignment.
  4. Test-Time Reward Modeling.
  5. Interpretable Alignment.
  6. Multi-Agent Alignment and Coordination.
  7. Problem Specification
  8. Alignment as a Bilevel Optimization Problem
  9. Proposed Method: ARCANE
  10. Rubric Representation and Decomposer Architecture
  11. Stakeholder–Decomposer Collaboration
  12. Training Data and Supervision
  13. Training-Time Optimization via Curriculum
  14. Stage I: Supervised Fine-Tuning (SFT).
  15. Stage II: GSPO over Rubric Proposals.
  16. ...and 33 more sections

Figures (1)

  • Figure 1: Test-Time Compute Scaling: SFT vs GSPO vs No Rubric and Oracle Baselines. Mean returns on the 44-task evaluation set under single-sample generation ($N{=}1$) and best-of-$N$ sampling ($N{=}1{\ldots}8$). Error bars show 95% bootstrap confidence intervals. The Oracle (Gold Rubric) baseline is an upper bound on rubric guidance effectiveness.