Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Memory-Based Reinforcement Learning Approach to Integrated Sensing and Communication

Homa Nikbakht, Michèle Wigger, Shlomo Shamai, H. Vincent Poor

TL;DR

The paper addresses joint sensing and communication over memoryful channels by formulating a capacity-distortion trade-off using Massey’s directed information and simplifying to unifilar channels. It recasts the waveform design as an online MDP and solves it with a deep deterministic policy gradient (DDPG) RL framework, enabling continuous actions that exploit channel memory. Key contributions include the online estimator design, the MDp formulation with a reward balancing information gain and sensing distortion, and numerical results showing substantial gains when the RL agent leverages an unbounded state-space. The work highlights the critical role of memory in ISAC and provides a scalable learning-based method to approach fundamental limits in memoryful ISAC systems.

Abstract

In this paper, we consider a point-to-point integrated sensing and communication (ISAC) system, where a transmitter conveys a message to a receiver over a channel with memory and simultaneously estimates the state of the channel through the backscattered signals from the emitted waveform. Using Massey's concept of directed information for channels with memory, we formulate the capacity-distortion tradeoff for the ISAC problem when sensing is performed in an online fashion. Optimizing the transmit waveform for this system to simultaneously achieve good communication and sensing performance is a complicated task, and thus we propose a deep reinforcement learning (RL) approach to find a solution. The proposed approach enables the agent to optimize the ISAC performance by learning a reward that reflects the difference between the communication gain and the sensing loss. Since the state-space in our RL model is à priori unbounded, we employ deep deterministic policy gradient algorithm (DDPG). Our numerical results suggest a significant performance improvement when one considers unbounded state-space as opposed to a simpler RL problem with reduced state-space. In the extreme case of degenerate state-space only memoryless signaling strategies are possible. Our results thus emphasize the necessity of well exploiting the memory inherent in ISAC systems.

A Memory-Based Reinforcement Learning Approach to Integrated Sensing and Communication

TL;DR

The paper addresses joint sensing and communication over memoryful channels by formulating a capacity-distortion trade-off using Massey’s directed information and simplifying to unifilar channels. It recasts the waveform design as an online MDP and solves it with a deep deterministic policy gradient (DDPG) RL framework, enabling continuous actions that exploit channel memory. Key contributions include the online estimator design, the MDp formulation with a reward balancing information gain and sensing distortion, and numerical results showing substantial gains when the RL agent leverages an unbounded state-space. The work highlights the critical role of memory in ISAC and provides a scalable learning-based method to approach fundamental limits in memoryful ISAC systems.

Abstract

In this paper, we consider a point-to-point integrated sensing and communication (ISAC) system, where a transmitter conveys a message to a receiver over a channel with memory and simultaneously estimates the state of the channel through the backscattered signals from the emitted waveform. Using Massey's concept of directed information for channels with memory, we formulate the capacity-distortion tradeoff for the ISAC problem when sensing is performed in an online fashion. Optimizing the transmit waveform for this system to simultaneously achieve good communication and sensing performance is a complicated task, and thus we propose a deep reinforcement learning (RL) approach to find a solution. The proposed approach enables the agent to optimize the ISAC performance by learning a reward that reflects the difference between the communication gain and the sensing loss. Since the state-space in our RL model is à priori unbounded, we employ deep deterministic policy gradient algorithm (DDPG). Our numerical results suggest a significant performance improvement when one considers unbounded state-space as opposed to a simpler RL problem with reduced state-space. In the extreme case of degenerate state-space only memoryless signaling strategies are possible. Our results thus emphasize the necessity of well exploiting the memory inherent in ISAC systems.

Paper Structure

This paper contains 11 sections, 3 theorems, 17 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Capacity-Distortion Tradeoff
  4. Reinforcement Learning Approach to ISAC
  5. MDP formulation of the capacity-distortion trade-off
  6. DDPG Algorithm
  7. Collecting Experience at the Agent
  8. Improving agents and environment networks
  9. Example: Binary Channel with Multiplicative Bernoulli State
  10. Numerical Analysis
  11. Conclusions

Key Result

Proposition 1

The capacity-distortion trade-off $\mathsf C (\mathsf{D})$ is where $g_i^*(\cdot, \cdot)$ is the argmin-estimator in eq:ustar.

Figures (4)

  • Figure 1: system model
  • Figure 2: Average reward as a function of $\beta$ with $n = 5000$ and $p = 0.1$ for degenerate, limited with $k \in \{0.1,0.4\}$ and unbounded state spaces.
  • Figure 3: Average reward as a function of $p$ and $\beta$ for $n = 5000$ for degenerate, limited with $k \in \{0.1,0.4\}$ and unbounded state spaces.
  • Figure 4: Communication gain versus the maximum distortion $\mathsf{D}$ with $p = 0.3$ for degenerate and unbounded state space cases.

Theorems & Definitions (5)

  • Definition 1
  • Proposition 1
  • Definition 2
  • Theorem 1
  • Theorem 2