Correction of Transformer-Based Models with Smoothing Pseudo-Projector

TL;DR

The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks without altering their core architecture and demonstrates consistent improvements in training behavior across a range of settings, with no adverse effects observed otherwise.

Abstract

The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks without altering their core architecture. It can be viewed as a hidden-representation corrector that reduces sensitivity to noise by suppressing directions induced by label-irrelevant input content. The design is inspired by the multigrid (MG) paradigm, originally developed to accelerate the convergence of iterative solvers for partial differential equations and boundary value problems, and later extended to more general linear systems through algebraic multigrid methods. We refer to the method as a pseudo-projector because its linear prototype corresponds to a strictly idempotent orthogonal projector, whereas the practical formulation employs learnable restriction and prolongation operators and therefore does not, in general, satisfy the properties of an exact orthogonal projection. We evaluate the proposed approach on transformer-based text classification tasks, as well as controlled synthetic benchmarks, demonstrating its effectiveness in improving training dynamics and robustness. Experimental results, together with supporting theoretical heuristics, indicate consistent improvements in training behavior across a range of settings, with no adverse effects observed otherwise. Our next step will be to extend this approach to language models.

Figures (25)

• Figure 1: Fig 1. “Wiggly curve" model structure
• Figure 2: Figure 2 Input parameters in test 1 including train and test sizes, batch size, hidden layers size, restriction (coarse) dimension, number of hidden layers, number of epochs and learning rate.
• Figure 3: Figure 3 compares the learned decision boundaries of the base model with and without the projector in test1.
• Figure 4: Figure 4 Validation metrics and losses with training history in test1.
• Figure 5: Figure 5 compares the learned decision boundaries of the base model with and without the projector in test2 with 800 and 200 training and test samples accordingly after 15 epochs in Test 2, where solid blue line shows learned boundaries.