Evaluating Dynamic Environment Difficulty for Obstacle Avoidance Benchmarking

TL;DR

This work tackles the lack of a quantitative metric for the difficulty of dynamic environments in obstacle avoidance. It designs six metrics—Obstacle Density, Traversability, Dynamic Traversability, VO Feasibility, Survivability, and Global Survivability—and validates them in a custom OpenAI Gym–style simulator that isolates environmental difficulty from perception and control errors, across two map datasets and multiple planners. The experiments (over 1.5 million trials) show that the Survivability metric yields the strongest monotonic relationship with planner success (SRCC $\approx$ 0.93) and low variability, making it especially suitable for fair benchmarking and guiding method refinement. The work also discusses practical use cases, including generating maps with predefined survivability and transferring survivability calculations to other simulators or real-world tests, with future work extending to 3D scenarios.

Abstract

Dynamic obstacle avoidance is a popular research topic for autonomous systems, such as micro aerial vehicles and service robots. Accurately evaluating the performance of dynamic obstacle avoidance methods necessitates the establishment of a metric to quantify the environment's difficulty, a crucial aspect that remains unexplored. In this paper, we propose four metrics to measure the difficulty of dynamic environments. These metrics aim to comprehensively capture the influence of obstacles' number, size, velocity, and other factors on the difficulty. We compare the proposed metrics with existing static environment difficulty metrics and validate them through over 1.5 million trials in a customized simulator. This simulator excludes the effects of perception and control errors and supports different motion and gaze planners for obstacle avoidance. The results indicate that the survivability metric outperforms and establishes a monotonic relationship between the success rate, with a Spearman's Rank Correlation Coefficient (SRCC) of over 0.9. Specifically, for every planner, lower survivability leads to a higher success rate. This metric not only facilitates fair and comprehensive benchmarking but also provides insights for refining collision avoidance methods, thereby furthering the evolution of autonomous systems in dynamic environments.

Figures (7)

• Figure 1: Dynamic environments with different difficulties. (a) shows a map with two walking pedestrians and one running pedestrian; (b) is with one walking pedestrian; (3) is with two running pedestrians. While it is intuitively clear that (b) represents a simpler environment than (a), determining whether (c) is simpler or more difficult than (a) remains unclear. The magnitude of the difficulty is hard to determine.
• Figure 2: The pipeline of the proposed custom simulator
• Figure 3: Visualization of one experiment trial in the custom simulator. green,36;blue,35,radius=3pt] (0,0) circle ;: Unobserved Obstacle; green,176;blue,210,radius=3pt] (0,0) circle ;: Observed Obstacle; green,240;blue,240] (0,0) rectangle ++(5pt,5pt);: Unexplored Area; green,200;blue,200] (0,0) rectangle ++(5pt,5pt);: Unoccupied Area
• Figure 4: Examples of metrics calculation. In (a), traversability is the average of traversable distances(shown by blue arrows) in 8 uniformly distributed directions at one sampled position. In (b), the white areas represent the infeasible VO regions. 4 velocity vectors(shown by blue arrows) are sampled at the sampled position. Only the velocity pointing right lies in the infeasible VO regions. So we have $n_{feasible}=3$ and $n_{infeasible}=1$. The resulting VO feasibility is thus $\frac{3}{4}$. In (c), a simple example of Survivability calculation is shown. The static robot is assumed to be placed in two positions. The static robot above does not intersect with any obstacle history trajectories in $T_{max}$ while the static robot below survives 2 seconds until it collides with the obstacle heading toward the left. The survivability is thus $(T_{max}+2)/2=(3+2)/2=2.5$
• Figure 5: All experiment results of the proposed metrics. In (a), the plot shows the relationship between the success rate and the pre-processed metrics for different planners. Different colors denote planners. The light-colored band surrounding the curve represents one standard deviation of the success rate of each planner under each level of the difficulty metric. In (b), the boxplot shows SRCC and CV values of different metrics.