Monotonicity as an Architectural Bias for Robust Language Models
Patrick Cooper, Alireza Nadali, Ashutosh Trivedi, Alvaro Velasquez
TL;DR
This work tackles the brittleness of large language models by proposing a structural inductive bias: monotonicity in Transformer feed-forward sublayers. By enforcing a monotone update $F(h) = h + A^\dagger g(Ah)$ with a fixed $A$ and a coordinatewise monotone $g$, while leaving attention unconstrained, the model maintains task performance while significantly improving adversarial robustness, demonstrated by a drop in gradient-based attack success from about $63$–$69\%$ to around $19\%$ and a large reduction in degradation under attacks. The approach yields a lossless distillation from pretrained models, preserves summarization quality within a small margin, and does not rely on data or objective changes or extra defenses. The results suggest that architectural constraints can meaningfully shape semantic refinement and perturbation propagation, offering a principled and scalable path toward more robust language models.
Abstract
Large language models (LLMs) are known to exhibit brittle behavior under adversarial prompts and jailbreak attacks, even after extensive alignment and fine-tuning. This fragility reflects a broader challenge of modern neural language models: small, carefully structured perturbations in high-dimensional input spaces can induce large and unpredictable changes in internal semantic representations and output. We investigate monotonicity as an architectural inductive bias for improving the robustness of Transformer-based language models. Monotonicity constrains semantic transformations so that strengthening information, evidence, or constraints cannot lead to regressions in the corresponding internal representations. Such order-preserving behavior has long been exploited in control and safety-critical systems to simplify reasoning and improve robustness, but has traditionally been viewed as incompatible with the expressivity required by neural language models. We show that this trade-off is not inherent. By enforcing monotonicity selectively in the feed-forward sublayers of sequence-to-sequence Transformers -- while leaving attention mechanisms unconstrained -- we obtain monotone language models that preserve the performance of their pretrained counterparts. This architectural separation allows negation, contradiction, and contextual interactions to be introduced explicitly through attention, while ensuring that subsequent semantic refinement is order-preserving. Empirically, monotonicity substantially improves robustness: adversarial attack success rates drop from approximately 69% to 19%, while standard summarization performance degrades only marginally.