Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor

Shu Yang, Jingyu Hu, Tong Li, Hanqi Yan, Wenxuan Wang, Di Wang

TL;DR

AutoMonitor-Bench introduces a ground-truth benchmark of 3,010 misbehavior-centric samples (with benign counterparts) to systematically evaluate LLM-based monitors across QA, code generation, and reasoning. It defines Miss Rate and False Alarm Rate to capture safety coverage and usability, and reveals substantial variability and a consistent MR–FAR trade-off across 22 models (12 proprietary, 10 open-source). A large-scale training corpus (153,581 samples) is used to fine-tune Qwen3-4B-Instruct, showing gains confined to trained misbehavior types and poor generalization to unseen strategies such as specification gaming. The work highlights that reliable, scalable misbehavior monitoring remains challenging and motivates future task-aware training and prompting strategies to improve robustness and transferability.

Abstract

We introduce AutoMonitor-Bench, the first benchmark designed to systematically evaluate the reliability of LLM-based misbehavior monitors across diverse tasks and failure modes. AutoMonitor-Bench consists of 3,010 carefully annotated test samples spanning question answering, code generation, and reasoning, with paired misbehavior and benign instances. We evaluate monitors using two complementary metrics: Miss Rate (MR) and False Alarm Rate (FAR), capturing failures to detect misbehavior and oversensitivity to benign behavior, respectively. Evaluating 12 proprietary and 10 open-source LLMs, we observe substantial variability in monitoring performance and a consistent trade-off between MR and FAR, revealing an inherent safety-utility tension. To further explore the limits of monitor reliability, we construct a large-scale training corpus of 153,581 samples and fine-tune Qwen3-4B-Instruction to investigate whether training on known, relatively easy-to-construct misbehavior datasets improves monitoring performance on unseen and more implicit misbehaviors. Our results highlight the challenges of reliable, scalable misbehavior monitoring and motivate future work on task-aware designing and training strategies for LLM-based monitors.

AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor

TL;DR

AutoMonitor-Bench introduces a ground-truth benchmark of 3,010 misbehavior-centric samples (with benign counterparts) to systematically evaluate LLM-based monitors across QA, code generation, and reasoning. It defines Miss Rate and False Alarm Rate to capture safety coverage and usability, and reveals substantial variability and a consistent MR–FAR trade-off across 22 models (12 proprietary, 10 open-source). A large-scale training corpus (153,581 samples) is used to fine-tune Qwen3-4B-Instruct, showing gains confined to trained misbehavior types and poor generalization to unseen strategies such as specification gaming. The work highlights that reliable, scalable misbehavior monitoring remains challenging and motivates future task-aware training and prompting strategies to improve robustness and transferability.

Abstract

We introduce AutoMonitor-Bench, the first benchmark designed to systematically evaluate the reliability of LLM-based misbehavior monitors across diverse tasks and failure modes. AutoMonitor-Bench consists of 3,010 carefully annotated test samples spanning question answering, code generation, and reasoning, with paired misbehavior and benign instances. We evaluate monitors using two complementary metrics: Miss Rate (MR) and False Alarm Rate (FAR), capturing failures to detect misbehavior and oversensitivity to benign behavior, respectively. Evaluating 12 proprietary and 10 open-source LLMs, we observe substantial variability in monitoring performance and a consistent trade-off between MR and FAR, revealing an inherent safety-utility tension. To further explore the limits of monitor reliability, we construct a large-scale training corpus of 153,581 samples and fine-tune Qwen3-4B-Instruction to investigate whether training on known, relatively easy-to-construct misbehavior datasets improves monitoring performance on unseen and more implicit misbehaviors. Our results highlight the challenges of reliable, scalable misbehavior monitoring and motivate future work on task-aware designing and training strategies for LLM-based monitors.
Paper Structure (29 sections, 10 figures, 5 tables)

This paper contains 29 sections, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Misbehaviors of frontier AI systems
  4. LLM-based Misbehavior Identification
  5. AutoMonitor-Bench
  6. Task Formulation
  7. Monitor input and output.
  8. Evaluation metrics.
  9. Benchmark Construction
  10. Data Statistics
  11. Fine-tuning LLM for Monitor Misbehavior
  12. Training dataset.
  13. Training details.
  14. Experiment setup
  15. Monitoring Input Formats
  16. ...and 14 more sections

Figures (10)

  • Figure 1: An illustrative example of LLM-based misbehavior monitoring. Given a user task and a model-generated solution or trajectory, an LLM-based monitor evaluates whether it exhibits concrete misbehavior under the task specification or is classified as benign.
  • Figure 2: Dataset examples across misbehavior categories. Misbehavior (red) and benign (green) responses illustrate insecure code generation, cue-driven reasoning errors, and evaluation gaming with fabricated citations.
  • Figure 3: AutoMonitor-Bench dataset statistics overview.
  • Figure 4: Training data distribution. The categories reasoning (thinking) and reasoning (no-thinking) indicate whether the solution or trajectory includes an explicit intermediate reasoning stage, i.e., reasoning traces enclosed in <think></think> tags.
  • Figure 5: Leaderboard of LLM-based monitors on AutoMonitor-Bench. Hatched bars indicate open-source models and solid bars indicate proprietary models. Red dashed lines denote the average performance across all evaluated models.
  • ...and 5 more figures