Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

O-TALC: Steps Towards Combating Oversegmentation within Online Action Segmentation

Matthew Kent Myers, Nick Wright, A. Stephen McGough, Nicholas Martin

TL;DR

The paper tackles the challenge of online temporal action segmentation for real-time human-robot interaction by introducing two backbone-invariant techniques: surround sampling during training to align training and online inference clips, and O-TALC, a real-time label-cleaning strategy to reduce oversegmentation. Surround sampling mitigates boundary misalignment and short-action omissions, while O-TALC explicitly removes short erroneous segments using static or class-based cutoffs, yielding improved F1 scores with minimal delay. Across datasets (CBAA, 50 Salads, Assembly-101), the approach achieves competitive online performance and, in some cases, matches offline models that rely on full temporal resolution. These contributions offer practical gains for real-time perception in HRI and suggest avenues for integrating longer-term temporal modeling with low-latency, buffer-friendly architectures.

Abstract

Online temporal action segmentation shows a strong potential to facilitate many HRI tasks where extended human action sequences must be tracked and understood in real time. Traditional action segmentation approaches, however, operate in an offline two stage approach, relying on computationally expensive video wide features for segmentation, rendering them unsuitable for online HRI applications. In order to facilitate online action segmentation on a stream of incoming video data, we introduce two methods for improved training and inference of backbone action recognition models, allowing them to be deployed directly for online frame level classification. Firstly, we introduce surround dense sampling whilst training to facilitate training vs. inference clip matching and improve segment boundary predictions. Secondly, we introduce an Online Temporally Aware Label Cleaning (O-TALC) strategy to explicitly reduce oversegmentation during online inference. As our methods are backbone invariant, they can be deployed with computationally efficient spatio-temporal action recognition models capable of operating in real time with a small segmentation latency. We show our method outperforms similar online action segmentation work as well as matches the performance of many offline models with access to full temporal resolution when operating on challenging fine-grained datasets.

O-TALC: Steps Towards Combating Oversegmentation within Online Action Segmentation

TL;DR

The paper tackles the challenge of online temporal action segmentation for real-time human-robot interaction by introducing two backbone-invariant techniques: surround sampling during training to align training and online inference clips, and O-TALC, a real-time label-cleaning strategy to reduce oversegmentation. Surround sampling mitigates boundary misalignment and short-action omissions, while O-TALC explicitly removes short erroneous segments using static or class-based cutoffs, yielding improved F1 scores with minimal delay. Across datasets (CBAA, 50 Salads, Assembly-101), the approach achieves competitive online performance and, in some cases, matches offline models that rely on full temporal resolution. These contributions offer practical gains for real-time perception in HRI and suggest avenues for integrating longer-term temporal modeling with low-latency, buffer-friendly architectures.

Abstract

Online temporal action segmentation shows a strong potential to facilitate many HRI tasks where extended human action sequences must be tracked and understood in real time. Traditional action segmentation approaches, however, operate in an offline two stage approach, relying on computationally expensive video wide features for segmentation, rendering them unsuitable for online HRI applications. In order to facilitate online action segmentation on a stream of incoming video data, we introduce two methods for improved training and inference of backbone action recognition models, allowing them to be deployed directly for online frame level classification. Firstly, we introduce surround dense sampling whilst training to facilitate training vs. inference clip matching and improve segment boundary predictions. Secondly, we introduce an Online Temporally Aware Label Cleaning (O-TALC) strategy to explicitly reduce oversegmentation during online inference. As our methods are backbone invariant, they can be deployed with computationally efficient spatio-temporal action recognition models capable of operating in real time with a small segmentation latency. We show our method outperforms similar online action segmentation work as well as matches the performance of many offline models with access to full temporal resolution when operating on challenging fine-grained datasets.
Paper Structure (8 sections, 3 figures, 4 tables, 1 algorithm)

This paper contains 8 sections, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction & Background
  2. Methodology
  3. Surround Sampling During Training
  4. O-TALC Algorithm
  5. Experiments
  6. Importance of Surround Sampling
  7. O-TALC Performance
  8. Conclusion

Figures (3)

  • Figure 1: An overview of our outlined online action segmentation approach utilising a surround sampled trained backbone model operating in a sliding window fashion with O-TALC.
  • Figure 2: F$_1$ performance vs. average class length on an extended sequence level for two models trained with dense and surround dense sampling on a real world CBAA dataset.
  • Figure 3: Example of O-TALC frame tidying on 50Salads split 1 (upper) and split 3 (lower) test sequences.