OpenTinker: Separating Concerns in Agentic Reinforcement Learning

TL;DR

OpenTinker is introduced, an infrastructure for reinforcement learning (RL) of large language model (LLM) agents built around a separation of concerns across algorithm design, execution, and agent-environment interaction that decomposes agentic learning systems into lightweight, composable components with clearly defined abstraction boundaries.

Abstract

We introduce OpenTinker, an infrastructure for reinforcement learning (RL) of large language model (LLM) agents built around a separation of concerns across algorithm design, execution, and agent-environment interaction. Rather than relying on monolithic, end-to-end RL pipelines, OpenTinker decomposes agentic learning systems into lightweight, composable components with clearly defined abstraction boundaries. Users specify agents, environments, and interaction protocols, while inference and training are delegated to a managed execution runtime. OpenTinker introduces a centralized scheduler for managing training and inference workloads, including LoRA-based and full-parameter RL, supervised fine-tuning, and inference, over shared resources. We further discuss design principles for extending OpenTinker to multi-agent training. Finally, we present a set of RL use cases that demonstrate the effectiveness of the framework in practical agentic learning scenarios.

Figures (5)

• Figure 1: OpenTinker architecture. The Scheduler orchestrates resource allocation, while Task Servers execute the training/inference jobs. The Client is a lightweight interface for definition and monitoring.
• Figure 2: Programming API with environment, clients, and scheduler.
• Figure 3: Distributed Multi-Agent Training. An agent protocol coordinator synchronizes multi-agent interactions between training/validation environments and agent instances running on a remote Task Server. The Task Server hosts multiple resource pools, each dedicated to specific agent tasks including trajectory rollout , forward inference (fwd) , and parameter updates.
• Figure 4: Synchronization mechanism of the Agent Protocol Coordinator. Global barriers synchronize the start and end of the rollout and update phases across turns. Within these phases, internal barriers manage the execution timing and order of individual agents.
• Figure 5: Functional validation of RL execution in OpenTinker. Each plot shows the evolution of validation metrics over training steps. Consistent improvement across all settings confirms correct reward propagation, trajectory handling, and policy optimization.