CosineGate: Semantic Dynamic Routing via Cosine Incompatibility in Residual Networks

TL;DR

This work tackles redundant computation in deep residual networks by introducing CosineGate, a dynamic residual routing mechanism that gates per-sample blocks based on the Cosine Incompatibility Ratio (CIR) between the identity and residual paths. CIR=1 - cos(phi(x), phi(F(x))) provides a self-supervised, geometry-based signal for semantic novelty, while gating is learned end-to-end via Gumbel-Softmax with a global progressive FLOPs regularization and a consistency term to stabilize training. The approach achieves a range of accuracy-efficiency trade-offs on CIFAR-10, including a Balanced setting that matches ResNet-20 accuracy with substantial FLOPs savings and a Conservative setting that surpasses ResNet-20 with minimal compute loss; MNIST validation further demonstrates strong redundancy detection. Overall, CIR-based gating offers a principled, lightweight, and biologically plausible pathway to enable efficient edge and TinyML deployment without auxiliary supervision or distillation.

Abstract

Modern deep residual networks perform substantial redundant computation by evaluating all residual blocks for every input, even when identity mappings suffice. We introduce CosineGate, an end-to-end differentiable architecture for dynamic routing in residual networks that uses cosine incompatibility between identity and residual feature representations as a self-supervised skip signal. CosineGate measures semantic redundancy through the Cosine Incompatibility Ratio (CIR), defined as 1 - cos(x, F(x)), and uses Gumbel-Softmax relaxation to enable per-sample, per-block gating during training. A progressive FLOPs regularization term controls average compute usage without destabilizing optimization. On CIFAR-10, CosineGate spans the accuracy-efficiency Pareto frontier: an aggressive configuration achieves 89.9 percent accuracy with 24.1 percent FLOPs savings, a balanced configuration achieves 91.3 percent accuracy with 28.5 percent savings at epoch 160, and a conservative configuration reaches a peak of 93.2 percent accuracy with minimal compute reduction. These results match or exceed ResNet-20 (91.3 percent) while reducing computation, without auxiliary supervision, distillation, or task-specific heuristics. Our results demonstrate that simple geometric measures of feature incompatibility provide a principled and effective signal for dynamic residual routing.

Figures (9)

• Figure 1: CosineGate signature view. Residual routing is driven by directional novelty. The angle $\theta$ between the identity path $x$ and residual update $F(x)$ defines CIR $=1-\cos(\theta)$, which controls the gate $g$. The output is $y=x+gF(x)$, enabling semantically grounded skipping when $F(x)$ aligns with $x$.
• Figure 2: Geometric intuition behind Cosine Incompatibility Ratio (CIR). When the residual vector $F_i(x_i)$ is directionally aligned with $x_i$, the update is redundant and can be skipped. Orthogonal residuals introduce new representational directions and should be computed.
• Figure 3: CosineGate routing pipeline. CIR and controller outputs define gate logits, which are relaxed via Gumbel-Softmax during training and thresholded at inference. The FLOPs penalty $\lambda_{\text{flops}}$ operates only at the loss level through $\overline{g}$ and influences gates indirectly via backpropagation.
• Figure 4: Network-level CosineGate architecture. Each residual block is augmented with an independent gate, enabling adaptive FLOPs allocation across network depth.
• Figure 5: Single CosineGate block. CIR measures directional novelty, Gumbel-Softmax enables differentiable routing, and hard thresholds ensure deterministic inference.