Integrating Semantic Communication and Human Decision-Making into an End-to-End Sensing-Decision Framework

TL;DR

The paper introduces a probabilistic end-to-end sensing-decision framework that couples semantic communication with human decision-making, enabling efficient task-oriented transmission. It builds on SINFONY and Information Maximization to optimize both semantic encoding and the presentation of information to a cognitive HDM agent modeled by a Generalized Context Model. Through simulations on tool wear, image, and audio datasets, the work demonstrates that balancing semantic detail with human cognitive capabilities can achieve accurate decisions with reduced bandwidth, latency, and power, though results depend on task context and expertise. The study also outlines open challenges, including practical optimization with black-box HDM, presentation design, and sender-receiver incentives, highlighting a path for future interdisciplinary development in semantic communication and human-in-the-loop systems.

Abstract

As early as 1949, Weaver defined communication in a very broad sense to include all procedures by which one mind or technical system can influence another, thus establishing the idea of semantic communication. With the recent success of machine learning in expert assistance systems where sensed information is wirelessly provided to a human to assist task execution, the need to design effective and efficient communications has become increasingly apparent. In particular, semantic communication aims to convey the meaning behind the sensed information relevant for Human Decision-Making (HDM). Regarding the interplay between semantic communication and HDM, many questions remain, such as how to model the entire end-to-end sensing-decision-making process, how to design semantic communication for the HDM and which information should be provided for HDM. To address these questions, we propose to integrate semantic communication and HDM into one probabilistic end-to-end sensing-decision framework that bridges communications and psychology. In our interdisciplinary framework, we model the human through a HDM process, allowing us to explore how feature extraction from semantic communication can best support HDM both in theory and in simulations. In this sense, our study reveals the fundamental design trade-off between maximizing the relevant semantic information and matching the cognitive capabilities of the HDM model. Our initial analysis shows how semantic communication can balance the level of detail with human cognitive capabilities while demanding less bandwidth, power, and latency.

Figures (7)

• Figure 1: Sketch of the end-to-end sensing-decision process for the example of tool wear assessment. It also situates the fundamental design trade-off between semantic communication and human decision-making.
• Figure 2: Block diagram of the end-to-end sensing-decision framework, i.e., the probabilistic system model including human decision-making.
• Figure 3: Comparison of the classification error rate of SINFONY with different number of channel uses $N_{\mathrm{Tx}}$ per encoder and central image processing with digital image transmission on the Tool Wear, Speech Commands, and UrbanSound8K -- Fold 10 validation dataset as a function of SNR.
• Figure 4: The simulated classification performance of the proposed end-to-end sensing-decision framework, including SINFONY and human decisions modeled by the GCM. Each column shows the accuracy on different datasets. From left to right: Tools, MNIST, and CIFAR10. The top row shows the accuracy as a function of SNR. The bottom row shows the accuracy as a function of the knowledge base size. Within each figure, the color of the lines indicates the number of features presented to the GCM.
• Figure 5: The simulated classification performance of the proposed end-to-end sensing-decision framework on the audio datasets Speech Commands and UrbanSound8K -- Fold 10. The top row shows the accuracy as a function of SNR. The bottom row shows the accuracy as a function of the knowledge base size. Within each figure, the color of the lines indicates the type and number of features presented to the GCM.

Theorems & Definitions (1)

• proof