Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks

Thibaut Boissin, Franck Mamalet, Valentin Lafargue, Mathieu Serrurier

TL;DR

Orthogonium tackles fragmentation in orthogonal and $1$-Lipschitz building blocks byProviding a unified PyTorch library with dense and convolutional layers that maintain strict Lipschitz constraints. The approach emphasizes native support for modern CNN features, efficient kernel implementations, and modular designs that enable fast exploration of hybrids like SOC, SLL, and AOL. Key contributions include a comprehensive API (OrthoLinear and orthogonal convolutions), extensive validation to catch subtle implementation errors, and performance parity with standard convolutions (roughly a 10% overhead on large-scale benchmarks). This work enables scalable, reliable experimentation for certifiably robust architectures across domains such as robust vision, normalizing flows, and stable recurrent networks, while open-sourcing the tool to foster community-driven verification and improvement.

Abstract

Orthogonal and 1-Lipschitz neural network layers are essential building blocks in robust deep learning architectures, crucial for certified adversarial robustness, stable generative models, and reliable recurrent networks. Despite significant advancements, existing implementations remain fragmented, limited, and computationally demanding. To address these issues, we introduce Orthogonium , a unified, efficient, and comprehensive PyTorch library providing orthogonal and 1-Lipschitz layers. Orthogonium provides access to standard convolution features-including support for strides, dilation, grouping, and transposed-while maintaining strict mathematical guarantees. Its optimized implementations reduce overhead on large scale benchmarks such as ImageNet. Moreover, rigorous testing within the library has uncovered critical errors in existing implementations, emphasizing the importance of standardized and reliable tools. Orthogonium thus significantly lowers adoption barriers, enabling scalable experimentation and integration across diverse applications requiring orthogonality and robust Lipschitz constraints. Orthogonium is available at https://github.com/deel-ai/orthogonium.

Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks

TL;DR

Orthogonium tackles fragmentation in orthogonal and -Lipschitz building blocks byProviding a unified PyTorch library with dense and convolutional layers that maintain strict Lipschitz constraints. The approach emphasizes native support for modern CNN features, efficient kernel implementations, and modular designs that enable fast exploration of hybrids like SOC, SLL, and AOL. Key contributions include a comprehensive API (OrthoLinear and orthogonal convolutions), extensive validation to catch subtle implementation errors, and performance parity with standard convolutions (roughly a 10% overhead on large-scale benchmarks). This work enables scalable, reliable experimentation for certifiably robust architectures across domains such as robust vision, normalizing flows, and stable recurrent networks, while open-sourcing the tool to foster community-driven verification and improvement.

Abstract

Orthogonal and 1-Lipschitz neural network layers are essential building blocks in robust deep learning architectures, crucial for certified adversarial robustness, stable generative models, and reliable recurrent networks. Despite significant advancements, existing implementations remain fragmented, limited, and computationally demanding. To address these issues, we introduce Orthogonium , a unified, efficient, and comprehensive PyTorch library providing orthogonal and 1-Lipschitz layers. Orthogonium provides access to standard convolution features-including support for strides, dilation, grouping, and transposed-while maintaining strict mathematical guarantees. Its optimized implementations reduce overhead on large scale benchmarks such as ImageNet. Moreover, rigorous testing within the library has uncovered critical errors in existing implementations, emphasizing the importance of standardized and reliable tools. Orthogonium thus significantly lowers adoption barriers, enabling scalable experimentation and integration across diverse applications requiring orthogonality and robust Lipschitz constraints. Orthogonium is available at https://github.com/deel-ai/orthogonium.
Paper Structure (42 sections, 1 theorem, 9 equations, 3 figures, 3 algorithms)

This paper contains 42 sections, 1 theorem, 9 equations, 3 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Motivations.
  3. Our Contributions.
  4. Dense Orthogonal Layers
  5. Unified API with OrthoLinear.
  6. Supported Orthogonalization Methods.
  7. Spectral Normalization.
  8. Flexibility and Extensibility.
  9. Implementation Efficiency.
  10. Orthogonal Convolutions
  11. Adaptive Orthogonal Convolution (AOC).
  12. Adaptive-SOC.
  13. Almost-Orthogonal Layers (AOL).
  14. SDP-based Lipschitz Layers (SLL).
  15. Sandwich-AOC.
  16. ...and 27 more sections

Key Result

Theorem 2.1

We can use the block convolution operator Block convolution operator allows to fuse the kernels of two convolutions to construct the kernel of a convolution equivalent to the composition of the two convolutions. It is defined in li_preventing_2019, and an efficient implementation is available in boi

Figures (3)

  • Figure 1: https://github.com/deel-ai/orthogonium/ offers a standardized API to use, and create 1-Lipschitz layers, allowing a user to construct, test, and improve easily such a kind of network
  • Figure 2: Implemented convolutional layers in https://github.com/deel-ai/orthogonium/. All run on GPU, accept stride, dilation, groups, padding_mode, and have (when possible) parity with nn.Conv2d.
  • Figure 3: The $\circledast$ can be used to enable $s\neq 1$ and $c_i \neq c_o$ configurations on SLL. The flexibility of the $\circledast$ allows for operations resulting in a block with a similar structure as the original ResNet block.

Theorems & Definitions (1)

  • Theorem 2.1: Explicit conv exponential