Live and Learn: Continual Action Clustering with Incremental Views

TL;DR

This work addresses continual action clustering under incremental camera views, a realistic challenge for multi-view clustering where all views are not available at once. It introduces CAC, which combines a category memory library with a consensus partition matrix, enabling never-ending knowledge transfer from historical views to new ones through a three-step alternate optimization. The model factorizes the history via $H^* = E B$ and updates $H_t^* = H_{t-1}^* + H_t W_t$ using targeted optimizations over $E$, $B$, and $W_t$, with proven convergence and $O(l m (n k^2 + n))$ time complexity. Experiments on six action datasets show that CAC outperforms 15 baselines in accuracy and efficiency, validating its effectiveness and scalability for continual MVC tasks.

Abstract

Multi-view action clustering leverages the complementary information from different camera views to enhance the clustering performance. Although existing approaches have achieved significant progress, they assume all camera views are available in advance, which is impractical when the camera view is incremental over time. Besides, learning the invariant information among multiple camera views is still a challenging issue, especially in continual learning scenario. Aiming at these problems, we propose a novel continual action clustering (CAC) method, which is capable of learning action categories in a continual learning manner. To be specific, we first devise a category memory library, which captures and stores the learned categories from historical views. Then, as a new camera view arrives, we only need to maintain a consensus partition matrix, which can be updated by leveraging the incoming new camera view rather than keeping all of them. Finally, a three-step alternate optimization is proposed, in which the category memory library and consensus partition matrix are optimized. The empirical experimental results on 6 realistic multi-view action collections demonstrate the excellent clustering performance and time/space efficiency of the CAC compared with 15 state-of-the-art baselines.

Figures (9)

• Figure 1: Framework of CAC. When a new view arrives, we only need to maintain a consensus partition matrix to achieve never-ending knowledge transfer between historical and new coming views with the category memory library.
• Figure 2: Exemplar frames in WVU dataset
• Figure 3: Impact of view order on CAC.
• Figure 4: Impact of view number on CAC.
• Figure 5: Various parameter $\lambda$ on CAC.