J.D. Peiffer

Video and wearable sensor data provide complementary information about human movement. Video provides a holistic understanding of the entire body in the world while wearable sensors, namely inertial measurement units (IMUs), provide high-resolution measurements of specific body segments. A robust method to fuse these modalities and obtain biomechanically accurate kinematics would have substantial utility for clinical assessment and monitoring. Although multiple video-sensor fusion methods exist, most assume that a time-intensive and often brittle sensor-body calibration process has already been completed. In this work, we employ an implicit function to combine handheld smartphone video and uncalibrated IMU data at their full temporal resolution. Our monocular, video-only, biomechanical reconstruction already performs well, with only 3.91 (1.55) degrees of mean adjusted angular error in knee kinematics across 60 recordings. Re-constructing from a fusion of video and IMU data reduces this error to 2.9 (1.27) degrees. We validate this method in a diverse group including individuals with no gait impairments, lower limb prosthesis users, and those with a history of stroke. We also show that IMU data allows accurate tracking through periods of visual occlusion, equivalent to video-only.

Markerless motion capture (MMC) is revolutionizing gait analysis in clinical settings by making it more accessible, raising the question of how to extract the most clinically meaningful information from gait data. In multiple fields ranging from image processing to natural language processing, self-supervised learning (SSL) from large amounts of unannotated data produces very effective representations for downstream tasks. However, there has only been limited use of SSL to learn effective representations of gait and movement, and it has not been applied to gait analysis with MMC. One SSL objective that has not been applied to gait is contrastive learning, which finds representations that place similar samples closer together in the learned space. If the learned similarity metric captures clinically meaningful differences, this could produce a useful representation for many downstream clinical tasks. Contrastive learning can also be combined with causal masking to predict future timesteps, which is an appealing SSL objective given the dynamical nature of gait. We applied these techniques to gait analyses performed with MMC in a rehabilitation hospital from a diverse clinical population. We find that contrastive learning on unannotated gait data learns a representation that captures clinically meaningful information. We probe this learned representation using the framework of biomarkers and show it holds promise as both a diagnostic and response biomarker, by showing it can accurately classify diagnosis from gait and is responsive to inpatient therapy, respectively. We ultimately hope these learned representations will enable predictive and prognostic gait-based biomarkers that can facilitate precision rehabilitation through greater use of MMC to quantify movement in rehabilitation.