High-Performance Self-Supervised Learning by Joint Training of Flow Matching

TL;DR

FlowFM addresses the cost and trade-off limitations of diffusion-based SSL by jointly training a representation encoder with a conditional velocity-field generator under flow matching. It decouples recognition and generation, enabling high-fidelity data synthesis alongside strong discriminative representations, and introduces Dynamic Guidance Switching to regularize the encoder. Empirically, FlowFM surpasses state-of-the-art SSL (SSL-Wearables) on five HAR datasets, reduces training time by approximately half, and achieves up to 51x faster inference while maintaining generative quality. This framework paves the way for efficient, on-device foundation models that deliver robust representations and controllable generation across modalities such as wearables and text-conditioned time-series generation.

Abstract

Diffusion models can learn rich representations during data generation, showing potential for Self-Supervised Learning (SSL), but they face a trade-off between generative quality and discriminative performance. Their iterative sampling also incurs substantial computational and energy costs, hindering industrial and edge AI applications. To address these issues, we propose the Flow Matching-based Foundation Model (FlowFM), which jointly trains a representation encoder and a conditional flow matching generator. This decoupled design achieves both high-fidelity generation and effective recognition. By using flow matching to learn a simpler velocity field, FlowFM accelerates and stabilizes training, improving its efficiency for representation learning. Experiments on wearable sensor data show FlowFM reduces training time by 50.4\% compared to a diffusion-based approach. On downstream tasks, FlowFM surpassed the state-of-the-art SSL method (SSL-Wearables) on all five datasets while achieving up to a 51.0x inference speedup and maintaining high generative quality. The implementation code is available at https://github.com/Okita-Laboratory/jointOptimizationFlowMatching.

Figures (6)

• Figure 1: Overall architecture of FlowFM during pre-training. The model consists of (left) Representation Encoder $f_\phi$ and (right) Velocity Field Network $v_\theta$. $f_\phi$ extracts representation $r$ from input $x_1$ and supplies it as a condition to $v_\theta$. $v_\theta$ takes point $x_t$ on the probability path and conditions ($r$, time $t$, text $y$) and is jointly trained to predict the target velocity field $u_t$. The DGS module improves robustness by preventing over-reliance on the representation by randomly masking $r$ during training.
• Figure 2: Text-to-Signal workflow: (1) In tuning, the network $v_{\theta}$ is fine-tuned conditioned on both representation $r$ and text embedding $y$ with frozen components. (2) In inference, the model generates signals $x'_1$ from noise $x_0$ via an ODE solver, conditioned solely on text $y$.
• Figure 3: Confusion matrices on HAR tasks: The left panel displays transfer learning results on the PAMAP2 dataset, while the right panel shows fine-tuning results on the REALWORLD dataset. The strong diagonal patterns demonstrate FlowFM's high classification accuracy across diverse activity classes.
• Figure 4: t-SNE visualization of learned representations: Comparison between raw data (left) and FlowFM representations (right) on PAMAP2 and REALWORLD datasets. The learned representations successfully disentangle activity clusters that overlap in the raw data space, demonstrating high-quality feature separation without supervision.
• Figure 5: Text-to-Signal generation examples: Generated 3-axis accelerometer signals (Samples) conditioned on text prompts are shown alongside reference data (References). The model accurately translates semantic descriptions—such as activity type, intensity, and subject characteristics—into realistic time-series patterns, capturing distinct motion signatures like "walking" versus "climbing stairs."