Time-Aware Policy Learning for Adaptive and Punctual Robot Control

TL;DR

Time-aware policy learning is introduced, a reinforcement learning framework that enables robots to explicitly perceive and reason with time as a first-class variable, providing a unified foundation for efficient, robust, resilient, and human-aligned robot autonomy.

Abstract

Temporal awareness underlies intelligent behavior in both animals and humans, guiding how actions are sequenced, paced, and adapted to changing goals and environments. Yet most robot learning algorithms remain blind to time. We introduce time-aware policy learning, a reinforcement learning framework that enables robots to explicitly perceive and reason with time as a first-class variable. The framework augments conventional reinforcement policies with two complementary temporal signals, the remaining time and a time ratio, which allow a single policy to modulate its behavior continuously from rapid and dynamic to cautious and precise execution. By jointly optimizing punctuality and stability, the robot learns to balance efficiency, robustness, resiliency, and punctuality without re-training or reward adjustment. Across diverse manipulation domains from long-horizon pick and place, to granular-media pouring, articulated-object handling, and multi-agent object delivery, the time-aware policy produces adaptive behaviors that outperform standard reinforcement learning baselines by up to 48% in efficiency, 8 times more robust in sim-to-real transfer, and 90% in acoustic quietness while maintaining near-perfect success rates. Explicit temporal reasoning further enables real-time human-in-the-loop control and multi-agent coordination, allowing robots to recover from disturbances, re-synchronize after delays, and align motion tempo with human intent. By treating time not as a constraint but as a controllable dimension of behavior, time-aware policy learning provides a unified foundation for efficient, robust, resilient, and human-aligned robot autonomy.

Figures (10)

• Figure 2: (continued)
• Figure 3: Time-aware policy learning pipeline.(A) A standard RL policy is first refined with a minimum-time objective to obtain the fastest feasible strategy, producing a temporal lower bound and an empirical upper bound on instability. (B) Remaining time $T^{\text{left}}$ and time-ratio $tr$ signals are then appended to the observation space, and the time-optimal policy is distilled into this augmented policy via behavior cloning to preserve high-speed behaviors. (C) The augmented policy is further trained with a time objective and an instability constraint, enabling adaptive execution across schedules while ensuring effective usage of the available time. (D) At inference, varying $tr$ modulates action tempo: larger $tr$ induces fast, momentum-exploiting behavior (e.g., throw-like placement), while smaller $tr$ yields slow, precise motions (e.g., careful stacking) without retraining.
• Figure 4: Time awareness improves efficiency and punctuality.(A) Real-world execution for three tasks. The vanilla policy performs sequential motions (approach-grasp-lift-place), whereas the time-aware policy leverages momentum and parallelizes phases (e.g., slide-while-grasp, swing-to-pour, and pre-tensioned drawer pull). (B) Simulation benchmarks across time ratio settings. Time used: completion time decreases as the time ratio increases, with the time-aware policy completing tasks faster than the vanilla baseline while adhering to scheduled durations. Time mismatch: the time-aware policy tightly tracks target timing with small completion-time mismatch. Instability: instability increases only when the time ratio is high (fast), remaining below the learned stability threshold when more time is available. Success rate: time-aware policy remain near 100% across tasks and schedules. Explicit temporal conditioning enables aggressive strategies when time is limited and cautious, stable behavior when time is abundant.
• Figure 6: (continued)
• Figure 7: Temporal observations modulate action generation and observation saliency. We analyze the effect of temporal variables during two manipulation stages: (A) grasping and (B) stacking. Left: current real-world snapshots. Middle: actions under modified temporal observations (other observations held constant) at the current state, visualized in simplified 3D space. Right: heatmaps depicting changes in observation importance under the same temporal observation variations (i.e., $\frac{\partial \pi(a \mid s, T, tr)}{\partial s}$). Within each stage, the upper row shows effects of different time ratio values; the lower row shows effects of different remaining time. The time-aware policy adapts action selection based on both time ratio and remaining time. Variations in these temporal observations modulate attention allocated to different states during action generation.