Attention to Entropic Communication

TL;DR

This work integrates attention with information theory by introducing relative attention entropy (RAE), a proper, bandwidth-aware generalization of relative entropy that employs attention weights to emphasize informative regions of the signal. It first establishes the uniqueness of relative entropy under analyticity, locality, properness, and calibration, then shows that naive weighting undermines properness; it then constructs attention as a renormalized, positive density $\mathcal{A}^{(w)}(s|I)$ and derives $\mathcal{D}_{s}^{(w)}(I_{A},I_{B})$, ensuring proper communication when sender and receiver have only partial knowledge of the receiver’s utility. Through both analytical derivations and illustrative examples (including misaligned vs aligned interests), the paper demonstrates how attention-weighted messaging can optimally shape a receiver’s actions, and how RAEs compare to other scoring rules, particularly in emphasizing high-attention regions. The results illuminate not only technical applications in Bayesian updating and compression but also socio-psychological implications for real-world communication and cooperation under misaligned incentives. Overall, the framework provides a principled route to design attentional, utility-aware communication protocols with provable properties and broad relevance.

Abstract

The concept of attention, numerical weights that emphasize the importance of particular data, has proven to be very relevant in artificial intelligence. Relative entropy (RE, aka Kullback-Leibler divergence) plays a central role in communication theory. Here we combine these concepts, attention and RE. RE guides optimal encoding of messages in bandwidth-limited communication as well as optimal message decoding via the maximum entropy principle (MEP). In the coding scenario, RE can be derived from four requirements, namely being analytical, local, proper, and calibrated. Weighted RE, used for attention steering in communications, turns out to be improper. To see how proper attention communication can emerge, we analyze a scenario of a message sender who wants to ensure that the receiver of the message can perform well-informed actions. If the receiver decodes the message using the MEP, the sender only needs to know the receiver's utility function to inform optimally, but not the receiver's initial knowledge state. In case only the curvature of the utility function maxima are known, it becomes desirable to accurately communicate an attention function, in this case a by this curvature weighted and re-normalized probability function. Entropic attention communication is here proposed as the desired generalization of entropic communication that permits weighting while being proper, thereby aiding the design of optimal communication protocols in technical applications and helping to understand human communication. For example, our analysis shows how to derive the level of cooperation expected under misaligned interests of otherwise honest communication partners.

Figures (4)

• Figure 1: Example of communication based on relative attention entropy and weighted relative entropy as discussed in Sect. \ref{['subsec:Attention-Example']}. Alice's bimodal knowledge state is given by the red solid curve. Bob's final knowledge state after Alice's communication is shown for various cases. The dashed black lines correspond to cases in which Alice uses relative attention entropy, and the dotted blue lines to cases she uses weighted relative entropy. Different results for the weight function $w(s)=\exp(\lambda\,s)$ with $\lambda=0$, $1$, $2$, $4,$$8$, $16$, and $32$ are shown from left two right, respectively. In case $\lambda=0$ relative entropy, relative attention entropy, and weighted relative entropy give the same result, the shown zero centered Gaussian. For $\lambda\ge8$ the different curves for the relative attention entropy results are visually indistinguishable and indicate the result of the $\lambda\rightarrow\infty$ limit.
• Figure 2: Attention functions corresponding to the cases $\lambda=0,\text{ }1,\text{ }2,\text{ and }4$ of Fig. \ref{['fig:Example-of-communication']} on logarithmic scale to display the unattended peak of the Alice's attention. Note that due to the strong exponential focus of the weights on larger $s$-values the attention peaks are displaced to the right w.r.t. the corresponding knowledge peaks.
• Figure 3: Sketch of the investigated communication scenario. Alice communicates parts of her knowledge to Bob about an unknown situation. After updating according to Alice's message Bob chooses his action, which for simplicity is here assumed to be a point in situation space $\mathcal{S}$ (for example, the blue point could indicate to which situation his action is best adapted). His action and the unknown situation determine Bob's resulting utility. Bob chooses his action by maximizing his expected utility given his knowledge after Alice informing him (blue equal probability contours of $\mathcal{P}(s|I_{\text{B}})$ in his mental copy of the situation space). The action and situation also determine a utility for Alice, which may or may not equal Bob's utility. Alice chooses her message such that her expected utility resulting from Bob's action is maximized in the light of her situation knowledge $\mathcal{P}(s|I_{\text{A}})$ (red contours). In case she is honest, she can only choose which parts of her knowledge she reveals with her message by deciding on a message topic $f(s)$; the message data is then determined to be $d=\langle f(s)\rangle_{(s|I_{\text{A}})}$.
• Figure 4: Knowledge states and preferred actions of Alice and Bob in case of misaligned interests before (left) and after (right) the communication. The plane of $s$-values is shown. Bob's knowledge state initially, $\mathcal{P}(s|I_{0})$ (left), and finally, $\mathcal{P}(s|I_{\text{B}})$ (right), is shown by the background color as well as by the blue contour lines at the 1- and 2-sigma levels. Alice's more precise knowledge is indicated only via red 1-, 2-, and 3-sigma level contours. The dots mark possible actions for Bob that are optimal for him under his knowledge (blue), under Alice's knowledge (green), or optimal for Alice (red). Comparing the two panels, especially the movement of Bob's optimal action (blue dot) between them, shows that Alice informs Bob such that he chooses an action that is a compromise between their interests.