CEC-MMR: Cross-Entropy Clustering Approach to Multi-Modal Regression

TL;DR

The paper tackles multi-modal regression where a single Gaussian or unimodal model is inadequate. It introduces CEC-MMR, which replaces the traditional MDN density estimation with a Cross-Entropy Clustering objective to automatically determine the number of Gaussian components and map data points to their underlying mode. By using a max-based conditional density and a pruning mechanism, CEC-MMR achieves competitive or superior performance on synthetic and real-world tasks while reducing manual hyperparameter tuning. This approach enhances interpretability and robustness in multi-modal regression, showing strong potential for applications requiring automatic component discovery and data-point to mode identification.

Abstract

In practical applications of regression analysis, it is not uncommon to encounter a multitude of values for each attribute. In such a situation, the univariate distribution, which is typically Gaussian, is suboptimal because the mean may be situated between modes, resulting in a predicted value that differs significantly from the actual data. Consequently, to address this issue, a mixture distribution with parameters learned by a neural network, known as a Mixture Density Network (MDN), is typically employed. However, this approach has an important inherent limitation, in that it is not feasible to ascertain the precise number of components with a reasonable degree of accuracy. In this paper, we introduce CEC-MMR, a novel approach based on Cross-Entropy Clustering (CEC), which allows for the automatic detection of the number of components in a regression problem. Furthermore, given an attribute and its value, our method is capable of uniquely identifying it with the underlying component. The experimental results demonstrate that CEC-MMR yields superior outcomes compared to classical MDNs.

Figures (4)

• Figure 1: Qualitative comparison between Gaussian Mixture Model (GMM) and Cross-Entropy Clustering (CEC) on a toy mouse-like dataset. The results presented were produced with the R packages mclust fraley2006mclust and CEC spurek2017r. The final clustering is illustrated with a variety of colors. It should be noted that in the case of CEC, the initial number of clusters (10) was reduced to 3. The presented example was inspired by tabor2014cross.
• Figure 2: Qualitative comparison between MDN and CEC-MMR (our) on a simple synthetic dataset, as discussed in bishop1994mixturepan2020implicit. The objective was to utilize ten Gaussian components to cover a 2D shape comprising two concentric circles (indicated with blue dots). For each regression mode, the final values of the mean and standard deviation parameters are presented in the form of a range plot. The results presented are those obtained after 1, 4, 16, 64, 512, and 1024 epochs of training (from left to right). It can be observed that both methods demonstrate comparable performance, but CEC-MMR exhibits a more rapid convergence in certain regions.
• Figure 3: Qualitative comparison between MDN and CEC-MMR (our) on a simple synthetic dataset, as discussed in bishop1994mixturepan2020implicit. The objective was to cover two 2D geometric shapes (indicated with blue dots), namely a zigzag (on the left) and an ellipse (on the right), using ten Gaussian components. For each regression mode, the final values of the mean and standard deviation parameters are presented in the form of a range plot. It can be observed that CEC-MMR achieves superior accuracy compared to MDN, which is particularly evident in the regions indicated by red rectangles (see their zoomed versions on the right). Furthermore, our method was capable of reducing the number of Gaussians to 9 (for the zigzag shape data) and 6 (for the ellipse shape data).
• Figure 4: Qualitative results of bimodal CEC-MMR (our) for the approximation of four 3D car shapes. The examples presented (indicated as blue dots) were generated by sampling 2048 points from the meshes of selected objects from the ShapeNet dataset chang2015shapenet. Each 3D object was treated as a function from $\mathbb{R}^2$ to $\mathbb{R}$. It should be noted that our method is capable of successfully modeling two complementary components, namely the car chassis and the car body.