Privileged Sensing Scaffolds Reinforcement Learning

TL;DR

This work tackles the challenge of training-time privileged sensing to improve policies that operate with limited test-time observations. It proposes Scaffolder, a model-based RL framework that scaffolds world models, critics, rewards, exploration, and representations using privileged observations during training, notably via a scaffolded world model and Nested Latent Imagination. The authors introduce the Sensory Scaffolding Suite (S3) of ten robotic tasks and demonstrate that Scaffolder delivers faster learning and higher final performance than baselines, often approaching or matching policies with test-time privileged sensing. The findings suggest that richer training-time sensing can substantially improve learning efficiency and policy quality, with practical impact for deploying robust agents in sensor-constrained settings. They also highlight areas for future work, including real-world robotics and deeper theoretical characterization of how privileged information shapes RL training signals.

Abstract

We need to look at our shoelaces as we first learn to tie them but having mastered this skill, can do it from touch alone. We call this phenomenon "sensory scaffolding": observation streams that are not needed by a master might yet aid a novice learner. We consider such sensory scaffolding setups for training artificial agents. For example, a robot arm may need to be deployed with just a low-cost, robust, general-purpose camera; yet its performance may improve by having privileged training-time-only access to informative albeit expensive and unwieldy motion capture rigs or fragile tactile sensors. For these settings, we propose "Scaffolder", a reinforcement learning approach which effectively exploits privileged sensing in critics, world models, reward estimators, and other such auxiliary components that are only used at training time, to improve the target policy. For evaluating sensory scaffolding agents, we design a new "S3" suite of ten diverse simulated robotic tasks that explore a wide range of practical sensor setups. Agents must use privileged camera sensing to train blind hurdlers, privileged active visual perception to help robot arms overcome visual occlusions, privileged touch sensors to train robot hands, and more. Scaffolder easily outperforms relevant prior baselines and frequently performs comparably even to policies that have test-time access to the privileged sensors. Website: https://penn-pal-lab.github.io/scaffolder/

Figures (15)

• Figure 1: Learning a policy to operate from partial observations can be aided by access to privileged sensors exclusively during training. Scaffolder improves world models, critics, exploration, and representation learning objectives to synthesize improved target policies.
• Figure 2: Sensory Scaffolding Suite (S3). We visualize four out of our ten diverse tasks, each exploring different restricted sensing scenarios such as proprioceptive-only inputs, noisy sensors, images, and occluded or moving viewpoints. We evaluate the enhancement of policy training using privileged sensors like multiple cameras, controllable cameras, object pose, and touch sensors. Refer to \ref{['supp:env_details']} for details on all environments.
• Figure 3: Scaffolder uses scaffolded observations to improve all components of training: world modelling, credit assignment, exploration, and policy representation.
• Figure 4: Nested Latent Imagination.
• Figure 5: The Sensory Scaffolding Suite (S3) of tasks. See \ref{['supp:env_details']} for details.