CDFlow: Building Invertible Layers with Circulant and Diagonal Matrices

TL;DR

CDFlow presents a circulant-diagonal factorization for invertible linear layers in normalizing flows, reducing parameter and computation costs while preserving expressiveness. By representing weight matrices as alternating diagonal and circulant factors and leveraging FFTs, the approach achieves $O(mn)$ storage, $O(mn)$ log-determinant evaluation, and $O(mn\log n)$ inversion, enabling efficient training and sampling. Integrated into a multi-scale flow with ActNorm and coupling, CDFlow delivers strong density estimation on standard image datasets and excels on data with periodic structure, such as galaxy images, while demonstrating favorable runtime characteristics. The work provides a practical, scalable path for high-dimensional flow-based modeling with structured matrices and FFT acceleration.

Abstract

Normalizing flows are deep generative models that enable efficient likelihood estimation and sampling through invertible transformations. A key challenge is to design linear layers that enhance expressiveness while maintaining efficient computation of the Jacobian determinant and inverse. We introduce a novel invertible linear layer based on the product of circulant and diagonal matrices. This decomposition reduces parameter complexity from $\mathcal{O}(n^2)$ to $\mathcal{O}(mn)$ using $m$ diagonal matrices and $m-1$ circulant matrices while still approximating general linear transformations. By leveraging the Fast Fourier Transform, our approach reduces the time complexity of matrix inversion from $\mathcal{O}(n^3)$ to $\mathcal{O}(mn\log n)$ and that of computing the log-determinant from $\mathcal{O}(n^3)$ to $\mathcal{O}(mn)$, where $n$ is the input dimension. We build upon this layer to develop Circulant-Diagonal Flow (CDFlow), which achieves strong density estimation on natural image datasets and effectively models data with inherent periodic structure. Furthermore, CDFlow significantly accelerates key operations in normalizing flows, providing practical benefits for scalable generative modeling.

Figures (11)

• Figure 1: Architecture of the CDFlow Model. The left portion shows a detailed schematic of the proposed CD-Convolution layer, which is constructed using circulant and diagonal matrices. The middle portion illustrates the internal structure of a flow module, where the input sequentially passes through an Actnorm layerkingma2018, the CD Convolution layer, and a coupling layerdinh2016. The right portion presents the overall CDFlow model architecture, in which the flow module is repeated K times and integrated into a multi-scale frameworkdinh2016 to effectively fuse channel-wise information.
• Figure 2: Ablation study on the hyperparameter $m$. Left: parameter counts (K) of the CD-Convolution layers and BPD on CIFAR-10. Middle: forward computation time (ms) across different channel sizes. Right: inverse computation time (ms) across different channel sizes.
• Figure 3: Comparison between the original image and outputs from different methods. For clearer and larger visualizations, please refer to Appendix \ref{['sec:gen_vis']}.
• Figure 4: Comparison of runtime (ms) for forward computation, inverse computation, and log-determinant calculation under varying numbers of channels, as well as inverse computation under different spatial sizes. The results represent the average of 100 runs. Our method consistently outperforms baselines, especially in inverse and log-determinant computations. The runtime of the Emerging method becomes excessively long at large spatial sizes and is therefore omitted from the corresponding figure.
• Figure 5: NLL comparison among flow-based models constructed with different linear layer types (F, L, U, LU, and DCD). Our DCD-type layer achieves performance close to the fully learnable F-type and surpasses the LU-type structure.