Computer Vision for Climbing

This Python application uses MediaPipe Pose from Google to give insights into climbing technique from just a video.

It can give metrics for limb extension, velocity of hands and feet, center of gravity, and a 3D pose graph.

GitHub

Center of Gravity Tracking


After experimenting with real climbing videos, center of gravity tracking turned out to be one of the most useful features. Being able to clearly see where the center of gravity is makes it obvious that most climbing techniques work by manipulating the center of gravity.


Volume Walking

It is well known that center of gravity placement is important with volume walking. Notice how each step is supported with a foot directly in line with the center of gravity. The impact of the center of gravity is even more apparent when you go frame by frame.


Pause video and use arrow keys to scrub by frame:
  • 0:01-0:02 Right leg doesn't extend until center of gravity is directly on top of the foot
  • 0:03-0:10 Transfer weight from leg to leg as he is walking across the volumes
  • 0:10-0:15 Centering weight under the finish hold to be able to match


    • Dynamic Slab Movement

    Being able to fall in a controlled way shows excellent proprioception. Here, James evenly balances his weight around the center line at the apex of the jump. Then he slowly falls over to catch the next hold.


    Watch at 0.25 speed:
  • 0:01-0:02 Loads weight and extends each leg as center of gravity passes by
  • 0:02-0:02 Balances around center line at apex of the jump
  • 0:02-0:03 Falls over to catch the next hold
  • 0:03-0:04 Uses hips to stop the barn door and momentum


    • Efficiency Analysis

    A prominent reason overhanging climbing is so hard is that it takes a lot of core strength to be able to place and keep the feet on the wall. On top of that, the arms can often be the only points of contact keeping you on. In order to limit the amount of energy spent per move, Naomi moves her center of gravity as little as possible.


    There's a lot going on here:
  • Power generation and roatational control with the legs before the campus move (more details in the next section)
  • Dissipates energy after catching the next hold by rotating her legs around the center of gravity
  • Efficiently brings foot up to the wall by maintaining a consistent center of gravity position

    • Limb Extension


    In this application, limb extension is calculated as the distance from the shoulder to hand and from the hip to foot. So far, this has been useful for showing power generation and rotation control.


    Power Generation and Rotation Control

    Here, Naomi uses leg extension to generate power for a campus move. At about 0:01, she straightens her legs and then brings her knees up to gain momentum. This can be seen in the graph as the "hump".


    Immediately after the hump in the graph, the two lines diverge. This is because she is controlling the rotation by extending her legs in opposite directions to prevent from spinning off of the wall.

    3D Pose Plot


    Using MediaPipe Pose allows for a 3D pose plot. This can be used to compare and understand body positions better.

    The origin is at the center of the hips and each red dot has coordinates (x, y, z) relative to the origin at the hip. These coordinates are available for every frame as well as coordinates relative to the frame size of the video