Rate-Distortion Signatures of Generalization and Information Trade-offs

TL;DR

This work introduces a rate-distortion-theoretic framework that treats stimulus-response behavior as an effective communication channel, derives rate-distortion (RD) frontiers from confusion matrices, and summarizes each system with two interpretable geometric signatures - slope and curvature - which capture the marginal cost and abruptness of accuracy-robustness trade-offs.

Abstract

Generalization to novel visual conditions remains a central challenge for both human and machine vision, yet standard robustness metrics offer limited insight into how systems trade accuracy for robustness. We introduce a rate-distortion-theoretic framework that treats stimulus-response behavior as an effective communication channel, derives rate-distortion (RD) frontiers from confusion matrices, and summarizes each system with two interpretable geometric signatures - slope ($β$) and curvature ($κ$) - which capture the marginal cost and abruptness of accuracy-robustness trade-offs. Applying this framework to human psychophysics and 18 deep vision models under controlled image perturbations, we compare generalization geometry across model architectures and training regimes. We find that both biological and artificial systems follow a common lossy-compression principle but occupy systematically different regions of RD space. In particular, humans exhibit smoother, more flexible trade-offs, whereas modern deep networks operate in steeper and more brittle regimes even at matched accuracy. Across training regimes, robustness training induces systematic but dissociable shifts in beta/kappa, revealing cases where improved robustness or accuracy does not translate into more human-like generalization geometry. These results demonstrate that RD geometry provides a compact, model-agnostic lens for comparing generalization behavior across systems beyond standard accuracy-based metrics.

Figures (3)

• Figure 1: Generalization curves and goodness-of-fit of rate--distortion predictions. (a) Empirical generalization $G(x,\hat{x})$ (points) plotted against inferred cost of error $\rho$ for humans and selected model families. Red curves show the corresponding exponential/RD-style prediction. Insets show observed vs. predicted off-diagonal confusion probabilities (RDT channel prediction), with RMSE reported per family. (b) Summary of exponential-gradient fits: observed binned generalization gradients versus predictions from the best-fit $a\,e^{-s d}$ model (red diagonal indicates equality). Each panel reports the fitted slope $s$ and RMSE, quantifying family-specific deviations from a simple exponential generalization law.
• Figure 2: Rate--distortion signatures across perturbation experiments and system families. (a) For each experiment/condition (x-axis; abbreviations as shown), points show systems’ RD summaries: normalized rate--distortion efficiency (AUC; left), $\log_{10}|\beta|$ (middle), and $\log_{10}|\kappa|$ (right), grouped by family/regime (legend). Values are computed per experiment$\times$condition block and visualized to highlight between-family structure across contexts. (b) Global view of the behavioral trade-off landscape: block-level $\beta_n$ (log- and z-normalized slope) vs. $\kappa_n$ (log- and z-normalized curvature), faceted by family/training-regime. Marker color encodes AUC. Across families, points form coherent bands with similar orientation but different intercepts, consistent with a shared accuracy--geometry coupling and family/regime-dependent location shifts in RD space.
• Figure 3: Effects of noise on generalization slopes across humans and models. Top panels show, for each system, the relationship between inferred error cost ($\rho$) and the log-probability of off-diagonal confusions, with colored lines denoting different uniform-noise levels. Each line’s slope defines $\beta$ for that noise level. Confusion counts are stabilized with an additive pseudocount ($\alpha=0.5$) before row-normalization for this visualization. Bottom panels plot $\beta$ as a function of noise level, revealing gradual severity dependence in humans and sharper, more noise-sensitive transitions in baseline CNNs, with specialised and all-noise networks exhibiting intermediate patterns.