Dedelayed: Deleting remote inference delay via on-device correction

TL;DR

Dedelayed addresses real-time video inference on resource-constrained devices by coupling a lightweight on-device model with a latency-aware remote model that predicts features for future frames. The remote component is conditioned on the measured delay via a delay embedding and provides high-level features $z_{t-\tau}$ that the local model fuses with the current frame to output $\hat{y}_t$, enabling timely predictions despite network latency. The system uses a joint training regime with a downlink autoencoder (DR-AE) to minimize bitrate while preserving task performance, demonstrated on real-time semantic segmentation for driving scenes under delays up to 167 ms. Results show Dedelayed outperforms both local-only and remote-only baselines, achieving gains comparable to a much larger model and delivering robust, deployable performance in dynamic networking conditions.

Abstract

Video comprises the vast majority of bits that are generated daily, and is the primary signal driving current innovations in robotics, remote sensing, and wearable technology. Yet, the most powerful video understanding models are too expensive for the resource-constrained platforms used in these applications. One approach is to offload inference to the cloud; this gives access to GPUs capable of processing high-resolution videos in real time. But even with reliable, high-bandwidth communication channels, the combined latency of video encoding, model inference, and round-trip communication prohibits use for certain real-time applications. The alternative is to use fully local inference; but this places extreme constraints on computational and power costs, requiring smaller models and lower resolution, leading to degraded accuracy. To address these challenges, we propose Dedelayed, a real-time inference system that divides computation between a remote model operating on delayed video frames and a local model with access to the current frame. The remote model is trained to make predictions on anticipated future frames, which the local model incorporates into its prediction for the current frame. The local and remote models are jointly optimized with an autoencoder that limits the transmission bitrate required by the available downlink communication channel. We evaluate Dedelayed on the task of real-time streaming video segmentation using the BDD100k driving dataset. For a round trip delay of 100 ms, Dedelayed improves performance by 6.4 mIoU compared to fully local inference and 9.8 mIoU compared to remote inference -- an equivalent improvement to using a model ten times larger.

Figures (7)

• Figure 1: Overview of various inference setups, including conventional local, remote, and split inference. Dedelayed combines a small on-device image model and a heavier cloud-based temporally predictive video model to produce accurate and on-time predictions.
• Figure 2: To demonstrate the effect of temporally predictive training, we train a 3D transformer to predict the next frame with an MSE loss on pixels. (a) shows the original video frame. (b) shows the difference between (a) and a future frame, with objects such as the traffic sign and road markings in different locations. (c) shows the pixel predictions of the 3D transformer. (d) shows the difference from the true future frame. While the predictive model cannot predict high-frequency details, it is able to accurately model the motion of objects, signs, and road markings.
• Figure 3: Example of activation maps from local and remote model components. The remote server uses the higher level of video detail to accurately distinguish and classify objects. The local model provides exact position adjustments based on the current frame. When making predictions from the combined activation map, small details that would be impossible to make out at low resolution (e.g., the distant pedestrians, labeled red) are accurately classified and localized.
• Figure 4: Time progresses left to right. The client-side camera produces video frames, which are sent across a communication network to the server. The server runs a heavyweight model using the latest video frame $x_{t - \tau}$ that it receives, in addition to a context of previously received video frames $x_{< t - \tau}$, as well as the measured delay $\tau$. This produces an output $z_{t - \tau}$ that the server sends to the client. The client pairs the latest received response $z_{t - \tau}$ with a freshly produced video frame $x_t$, and runs these inputs through a lightweight model. This finally produces a timely result $\hat{y}_t$ that can be used in real-time delay-sensitive applications.
• Figure 5: Dedelayed workflow. The complementary properties of local and remote inference are exploited to produce real-time streaming video understanding that is both accurate and on time.