Consistency Deep Equilibrium Models

TL;DR

This paper tackles the latency of implicit Deep Equilibrium Models (DEQs) by introducing Consistency Deep Equilibrium Models (C-DEQ). By reframing the DEQ solving process as a fixed-point ODE (FP-ODE) trajectory, the authors distill a consistency mapping that directly maps intermediate solver states to the equilibrium, enabling few-step or even single-step inference while preserving teacher performance. The approach integrates an Anderson Acceleration (AA) informed parameterization and a dual-consistency distillation objective (global and local) plus a task regularizer to stabilize training and support multi-step refinement. Extensive experiments across language modeling, image classification, and graph node classification demonstrate 2-20× consistency gains over implicit DEQs under the same few-step budget, with competitive to state-of-the-art explicit baselines and reduced latency. The work offers a practical pathway to deploy fixed-point implicit models in real-time and resource-constrained settings, significantly narrowing the efficiency gap between implicit and explicit architectures.

Abstract

Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage. However, DEQs incur significant inference latency due to the iterative nature of fixed-point solvers. In this work, we introduce the Consistency Deep Equilibrium Model (C-DEQ), a novel framework that leverages consistency distillation to accelerate DEQ inference. We cast the DEQ iterative inference process as evolution along a fixed ODE trajectory toward the equilibrium. Along this trajectory, we train C-DEQs to consistently map intermediate states directly to the fixed point, enabling few-step inference while preserving the performance of the teacher DEQ. At the same time, it facilitates multi-step evaluation to flexibly trade computation for performance gains. Extensive experiments across various domain tasks demonstrate that C-DEQs achieves consistent 2-20$\times$ accuracy improvements over implicit DEQs under the same few-step inference budget.

Figures (3)

• Figure 1: Illustration of trajectory independence of DEQs and consistency mapping of C-DEQs. (a) Trajectories of a path-independent DEQ. Various initial states (shaded region) and solvers can yield a manifold of a set of trajectories (dashed) toward the fixed point. We propose to fix the initial point and the solver to select a unique trajectory (red) converging to $({\bm{z}}_T, T)$. (b) Consistency mapping of a C-DEQ. C-DEQ trains a consistency model $h_{\boldsymbol{\phi}}$ to map (green) any intermediate states along the DEQ trajectory (e.g., $({\bm{z}}_t, t)$ and $({\bm{z}}_{t'}, t')$) directly to the equilibrium state.
• Figure 2: We visualize the residual trajectory across iteration steps $K$ on four benchmarks. Compared to standard DEQ (yellow) and previous acceleration methods like HyperDEQ/IGNN-Solver (green), our C-DEQ (red) consistently improves the convergence path and achieves lower residuals in fewer steps. Notably, on graph tasks (\ref{['subfig:arxiv']}, \ref{['subfig:products']}) with theoretically well-posed backbones, C-DEQ reaches near-equilibrium almost instantaneously.
• Figure 3: Ablative studies on C-DEQ. By explicitly leveraging solver history, C-DEQ (red) approaches the converged baseline (dotted) in only 5--6 steps, significantly faster than the non-AA variant (green) which requires nearly $2\times$ the steps.

Theorems & Definitions (1)

• Definition 3.1: Anderson acceleration