A Framework for Semantics-based Situational Awareness during Mobile Robot Deployments

TL;DR

The paper presents a semantics-based framework for situational awareness in mobile robot deployments, introducing environment semantic indicators (laser noise, risk, SHA, radiation) and the Situational Semantics Richness (SSR) metric to aggregate semantic information. Indicators are computed with explicit formulas and normalized to [0,1], then combined via weighted sums with exponential weights to yield $R$ and $R_{norm}$, while an attention-based temporal smoothing refines SSR over time. The approach emphasizes explainability and expert-driven tunability over purely data-driven models, aiming to improve HRT planning and decision-making in disaster response scenarios. Experimental validation on a Jackal platform demonstrates SSR’s sensitivity to semantic changes and its ability to reflect complex, multi-modal semantic richness in real-time, supporting scalable and interpretable SA for safety-critical operations.

Abstract

Deployment of robots into hazardous environments typically involves a ``Human-Robot Teaming'' (HRT) paradigm, in which a human supervisor interacts with a remotely operating robot inside the hazardous zone. Situational Awareness (SA) is vital for enabling HRT, to support navigation, planning, and decision-making. This paper explores issues of higher-level ``semantic'' information and understanding in SA. In semi-autonomous, or variable-autonomy paradigms, different types of semantic information may be important, in different ways, for both the human operator and an autonomous agent controlling the robot. We propose a generalizable framework for acquiring and combining multiple modalities of semantic-level SA during remote deployments of mobile robots. We demonstrate the framework with an example application of search and rescue (SAR) in disaster response robotics. We propose a set of ``environment semantic indicators" that can reflect a variety of different types of semantic information, e.g. indicators of risk, or signs of human activity, as the robot encounters different scenes. Based on these indicators, we propose a metric to describe the overall situation of the environment called ``Situational Semantic Richness (SSR)". This metric combines multiple semantic indicators to summarise the overall situation. The SSR indicates if an information-rich and complex situation has been encountered, which may require advanced reasoning for robots and humans and hence the attention of the expert human operator. The framework is tested on a Jackal robot in a mock-up disaster response environment. Experimental results demonstrate that the proposed semantic indicators are sensitive to changes in different modalities of semantic information in different scenes, and the SSR metric reflects overall semantic changes in the situations encountered.

Figures (5)

• Figure 1: Mobile robot semantic SA.
• Figure 2: Semantics-based SA framework: yellow box refers to low-level semantics, orange box refers to high-level semantics and red box refers to context. Black dash lines indicate the potential connections among different levels.
• Figure 3: (a): Layout for Experiment I. The dark blue box area (scenario 1) on the ground is used for laser noise or the radiation source (uranium rock). The yellow box area (scenario 2) is used for hazmat signs or personal belongings; (b): Layout for two environment semantics scenarios of experiment II; (c): Layout for three environment semantics scenarios of experiment II. In the picture, some environment semantics are covered by red barriers.
• Figure 4: SSR and environment semantics intensity timeline. Zones between dash lines refer to the corresponding semantics detected from the environment.
• Figure 5: SSR and environment semantics intensity timeline