Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

Daocheng Fu, Jianbiao Mei, Rong Wu, Xuemeng Yang, Jia Xu, Ding Wang, Pinlong Cai, Yong Liu, Licheng Wen, Botian Shi

TL;DR

This paper introduces Trainee-Bench, a dynamic benchmarking environment that simulates a workplace where an agent must schedule streaming tasks, actively explore under partial observability, and continually evolve by distilling generalized strategies from prior tasks. By formalizing a state-transition environment, diverse rule-based meta-tasks, partial observability, and composite task streams with automated verification, the framework assesses three core capabilities: dynamic scheduling, prudent exploration, and continual learning. Experimental results show current SOTA LLM agents struggle in dynamic, uncertain settings, with limited gains from internal experience and substantial benefits from human guidance, underscoring a reliability gap for production use. Trainee-Bench thus provides a rigorous, production-oriented benchmark to drive research toward robust exploration, adaptive scheduling, and lifelong learning in intelligent agents.

Abstract

The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets performance upper bounds in static environments, overlooking robustness for stochastic real-world deployment. We identify three key challenges: dynamic task scheduling, active exploration under uncertainty, and continuous learning from experience. To bridge this gap, we introduce \method{}, a dynamic evaluation environment that simulates a "trainee" agent continuously exploring a novel setting. Unlike traditional benchmarks, \method{} evaluates agents along three dimensions: (1) context-aware scheduling for streaming tasks with varying priorities; (2) prudent information acquisition to reduce hallucination via active exploration; and (3) continuous evolution by distilling generalized strategies from rule-based, dynamically generated tasks. Experiments show that cutting-edge agents have significant deficiencies in dynamic environments, especially in active exploration and continual learning. Our work establishes a framework for assessing agent reliability, shifting evaluation from static tests to realistic, production-oriented scenarios. Our codes are available at https://github.com/KnowledgeXLab/EvoEnv

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

TL;DR

This paper introduces Trainee-Bench, a dynamic benchmarking environment that simulates a workplace where an agent must schedule streaming tasks, actively explore under partial observability, and continually evolve by distilling generalized strategies from prior tasks. By formalizing a state-transition environment, diverse rule-based meta-tasks, partial observability, and composite task streams with automated verification, the framework assesses three core capabilities: dynamic scheduling, prudent exploration, and continual learning. Experimental results show current SOTA LLM agents struggle in dynamic, uncertain settings, with limited gains from internal experience and substantial benefits from human guidance, underscoring a reliability gap for production use. Trainee-Bench thus provides a rigorous, production-oriented benchmark to drive research toward robust exploration, adaptive scheduling, and lifelong learning in intelligent agents.

Abstract

The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets performance upper bounds in static environments, overlooking robustness for stochastic real-world deployment. We identify three key challenges: dynamic task scheduling, active exploration under uncertainty, and continuous learning from experience. To bridge this gap, we introduce \method{}, a dynamic evaluation environment that simulates a "trainee" agent continuously exploring a novel setting. Unlike traditional benchmarks, \method{} evaluates agents along three dimensions: (1) context-aware scheduling for streaming tasks with varying priorities; (2) prudent information acquisition to reduce hallucination via active exploration; and (3) continuous evolution by distilling generalized strategies from rule-based, dynamically generated tasks. Experiments show that cutting-edge agents have significant deficiencies in dynamic environments, especially in active exploration and continual learning. Our work establishes a framework for assessing agent reliability, shifting evaluation from static tests to realistic, production-oriented scenarios. Our codes are available at https://github.com/KnowledgeXLab/EvoEnv
Paper Structure (40 sections, 4 equations, 5 figures, 5 tables, 1 algorithm)

This paper contains 40 sections, 4 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Trainee-Bench
  3. Preliminaries
  4. Meta-Task Design
  5. Dynamic Task Instantiation.
  6. Task Diversity and Coverage.
  7. Partial Observability.
  8. Dynamic Composite Scenarios
  9. Conflict Resolution:
  10. Temporal Prioritization:
  11. Inter-task Dependencies:
  12. Automated Verification Mechanism
  13. Interaction Protocol
  14. Benchmark Design Characteristics
  15. Robustness against Memorization.
  16. ...and 25 more sections

Figures (5)

  • Figure 1: Overview of challenges in current agent systems: (1) effective scheduling and multi-task planning for streaming task inputs; (2) the ability to suspect unsolvable tasks and solicit guidance through active exploration; and (3) robust experience summarization, retrieval, and utilization to enhance performance stability.
  • Figure 2: Overview of Trainee-Bench construction: (1) Unique task instances generated via random parameters. (2) Temporal composition of instances into task streams. (3) Automated verification of execution results.
  • Figure 3: Formulation for the interaction among the environment, agent, and MLLM service.
  • Figure 4: Comparison contrasting the upper bound of the agent's autonomous capability against the performance achieved through human-provided clues.
  • Figure 5: Screenshot of the human-AI collaboration interaction interface: The left side shows the role settings and task description, the middle is the tools page, and different toolsets can be selected by switching tabs at the top. The right side is the evaluation page, where you can view the scores.