Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Machine Learning for move sequence visualization and generation in climbing

Thomas Rimbot, Martin Jaggi, Luis Barba

TL;DR

This paper investigates applying machine learning to the sport of bouldering, focusing on move sequence visualization and the prediction of move order from holds. Building on prior move sequence detection work, it develops a visualization pipeline that reconstructs a climbing motion as a 99-landmark skeleton and renders it on a wall image. It then experiments with three Transformer-based approaches to translate holds sequences into move sequences, including a seq2seq model, an autoregressive transformer with positional embeddings, and a simplified transformer. Results are preliminary and reveal data and modeling challenges, but establish a practical baseline and roadmap for improving visualization fidelity and predictive performance in future work.

Abstract

In this work, we investigate the application of Machine Learning techniques to sport climbing. Expanding upon previous projects, we develop a visualization tool for move sequence evaluation on a given boulder. Then, we look into move sequence prediction from simple holds sequence information using three different Transformer models. While the results are not conclusive, they are a first step in this kind of approach and lay the ground for future work.

Using Machine Learning for move sequence visualization and generation in climbing

TL;DR

This paper investigates applying machine learning to the sport of bouldering, focusing on move sequence visualization and the prediction of move order from holds. Building on prior move sequence detection work, it develops a visualization pipeline that reconstructs a climbing motion as a 99-landmark skeleton and renders it on a wall image. It then experiments with three Transformer-based approaches to translate holds sequences into move sequences, including a seq2seq model, an autoregressive transformer with positional embeddings, and a simplified transformer. Results are preliminary and reveal data and modeling challenges, but establish a practical baseline and roadmap for improving visualization fidelity and predictive performance in future work.

Abstract

In this work, we investigate the application of Machine Learning techniques to sport climbing. Expanding upon previous projects, we develop a visualization tool for move sequence evaluation on a given boulder. Then, we look into move sequence prediction from simple holds sequence information using three different Transformer models. While the results are not conclusive, they are a first step in this kind of approach and lay the ground for future work.

Paper Structure

This paper contains 18 sections, 1 equation, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Move sequence detection
  3. Pose estimation
  4. Static extremity detection
  5. Clustering and visualization
  6. Move sequence visualization
  7. Skeleton generation
  8. Generation of the landmarks sequence for the extremities
  9. Generation of the landmarks sequence for the rest of the body
  10. Selection interface
  11. Move sequence prediction
  12. First try: adaptation of a Sequence to Sequence network
  13. Next try: Autoregressive Transformer model with Positional Embedding
  14. Data formatting
  15. Model specifics
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Move sequence detection from move_seq_det
  • Figure 2: Generated skeleton from a move sequence, using the procedure described in \ref{['ssec:skeleton_generation']}
  • Figure 3: Selection interface with examples of move sequence (left) and holds sequence (right)
  • Figure 4: Seq2seq model loss trend as a function of the epoch
  • Figure 5: Seq2seq move sequence prediction (left) against the truth (right)
  • ...and 2 more figures