Millimeter-Wave Gesture Recognition in ISAC: Does Reducing Sensing Airtime Hamper Accuracy?

TL;DR

This work addresses how reducing sensing airtime in mmWave ISAC affects gesture recognition. It collects a high-quality dataset using two devices sweeping $(50\times56)$ beam pairs and trains a CNN on input tensors with dimensions $(t, tx, rx)$, then systematically simulates reduced sensing airtime via subsampling. The results show that about $25\%$ sensing airtime barely impacts accuracy ($\approx0.15$ percentage points) when subsampling is performed in space (beam pairs) and balanced across $tx$ and $rx$, while temporal subsampling degrades performance more noticeably. The findings highlight that mmWave ISAC can achieve ultra-high throughput alongside accurate sensing for XR and in-home applications, supported by a public dataset and codebase.

Abstract

Most Integrated Sensing and Communications (ISAC) systems require dividing airtime across their two modes. However, the specific impact of this decision on sensing performance remains unclear and underexplored. In this paper, we therefore investigate the impact on a gesture recognition system using a Millimeter-Wave (mmWave) ISAC system. With our dataset of power per beam pair gathered with two mmWave devices performing constant beam sweeps while test subjects performed distinct gestures, we train a gesture classifier using Convolutional Neural Networks. We then subsample these measurements, emulating reduced sensing airtime, showing that a sensing airtime of 25 % only reduces classification accuracy by 0.15 percentage points from full-time sensing. Alongside this high-quality sensing at low airtime, mmWave systems are known to provide extremely high data throughputs, making mmWave ISAC a prime enabler for applications such as truly wireless Extended Reality.

Figures (6)

• Figure 1: The end-to-end system, going from power per beam pair measurements to gesture classification.
• Figure 2: The experimental setup, as seen from the test subject's point of view. The transmitting and receiving devices are placed side-by-side and both aimed towards the test subject.
• Figure 3: Classification accuracy for different subsampling factors after 100 epochs. For each datapoint, training was repeated 25 times, with mean and $\pm$1 standard deviation visualized. Slight horizontal offsets are introduced during visualization to make error bars more distinguishable (all correct values for the subsampling factor are integers). Subsampling factor 1 reverts to the baseline (100% sensing airtime) for each approach, indicated with a purple star and horizontal dashed line at baseline performance level.
• Figure 4: Two arbitrarily selected confusion matrices showing that, even with severe subsampling, the classifier performs well for all classes.
• Figure 5: The classification accuracy results from Fig. \ref{['fig:performance_100']} re-plotted with x-axis values adjusted to show the actual rather than target subsampling factor (now without horizontal offset) after 100 epochs. The brown line and axes show the airtime fractions for sensing and communications. The communications axis is kept linear to emphasize how rapidly it increases with increasing sensing subsampling. The horizontal baseline performance line is removed for visual clarity.