The Sandbox Environment for Generalizable Agent Research (SEGAR)

TL;DR

The Sandbox Environment for Generalizable Agent Research (SEGAR) improves the ease and accountability of generalization research in RL, as generalization objectives can be easy designed by specifying task distributions, which in turns allows the researcher to measure the nature of the generalization objective.

Abstract

A broad challenge of research on generalization for sequential decision-making tasks in interactive environments is designing benchmarks that clearly landmark progress. While there has been notable headway, current benchmarks either do not provide suitable exposure nor intuitive control of the underlying factors, are not easy-to-implement, customizable, or extensible, or are computationally expensive to run. We built the Sandbox Environment for Generalizable Agent Research (SEGAR) with all of these things in mind. SEGAR improves the ease and accountability of generalization research in RL, as generalization objectives can be easy designed by specifying task distributions, which in turns allows the researcher to measure the nature of the generalization objective. We present an overview of SEGAR and how it contributes to these goals, as well as experiments that demonstrate a few types of research questions SEGAR can help answer.

Figures (10)

• Figure 1: SEGAR at a glance. SEGAR begins with design of tasks, which is done by defining factor and entity types (collections of factor types), then using these types to define rules for the transition function and the other components of the task. After agents are trained on these tasks, this design process enables the researcher to perform detailed analysis on the agents' generalization capabilities.
• Figure 2: The state space is a product of two vector spaces, each corresponding to an entity. Each entity contains a different set of factor types, which make up the basis of each entity vector space, and the state space is not structured beyond being a vector space and the basis ordering fixed by the environment, and it can be of arbitrary size, depending on the numbers and types of entities.
• Figure 3: (a) Generalization performance gap of a PPO agent as a function of the sample-based Wasserstein-2 distance between latent factors of $n$ random train tasks and 100 test tasks, aggregated by task type and difficulty. (b) Generalization performance gap of a PPO agent as a function of the sample-based Wasserstein-2 distance between latent factors of $n$ random train tasks and 100 test tasks, split by task type and difficulty. Each point represents the performance of an agent trained on a different number of levels $n$. Trend lines show the least-squares line of best fit, together with 95% confidence intervals, Spearman $\rho$ coefficient and $p$-values.
• Figure 4: (a) Generalization performance gap of a PPO agent as a function of the sample-based Wasserstein-2 distance between latent factors of $n$ random train tasks and 100 test tasks, split by task type and difficulty. The test distribution has a higher difficulty level than the corresponding training distribution (easy $\to$ medium, medium $\to$ hard). Generalization gap increases with higher $W_2$ values. (b) Generalization performance gap of a PPO agent as a function of the lower-bound on mutual information between learned state representations and latent environment factors. Each point represents the performance of an agent trained on a different number of levels $n$. Trend lines show the least-squares line of best fit, together with 95% confidence intervals, Spearman $\rho$ coefficient and $p$-values. Smaller generalization gap doesn't necessarily require better information on latent factors.
• Figure 5: (a) Generalization performance gap of a PPO agent as a function of the mutual information between learned state representations and latent environment factors, when adding popular self-supervised objectives. Each point represents the performance of an agent trained on a different number of levels $n$. Trend lines show the least-squares line of best fit, together with 95% confidence intervals, Spearman $\rho$ coefficient and $p$-values. Self-supervised signals can improve information about latent factors, and the generalization gap. (b) Performance of a PPO agent as a function of training frames. Each curve corresponds to a different task, difficulty and number of levels configuration. Agent's performance depends mostly on diversity of training tasks.