publications | J.D. Peiffer

2026

Portable biomechanics laboratory enables clinically accessible movement analysis from a handheld smartphone

J.D. Peiffer, Kunal Shah, Irina Djuraskovic, and 6 more authors

npj Digital Medicine, Jun 2026

Abs DOI Video Website

Movement directly reflects neurological and musculoskeletal health, yet objective biomechanical assessment is rarely available in routine care. We introduce Portable Biomechanics Laboratory (PBL), a platform for fitting biomechanical models to handheld smartphone video. We validate PBL on over 15 hours of data synchronized to ground truth motion capture, finding joint-angle errors \textless 3∘ across patients with neurological injury, lower-limb prosthesis users, pediatric inpatients, and controls. Across 1021 videos recorded in prospective clinical deployment, PBL was easy to implement, yielded reliable gait metrics (ICC \textgreater 0.9), and detected clinically relevant differences in movement. For cervical myelopathy patients, its gait quality measures correlated with modified Japanese Orthopedic Association (mJOA) scores and were responsive to clinical intervention. Handheld smartphone video can therefore deliver accurate, scalable, and low-burden biomechanical measurement, enabling greatly increased monitoring of movement impairments. We release the first clinically validated method for measuring whole-body kinematics from handheld smartphone video at https://IntelligentSensingAndRehabilitation.github.io/MonocularBiomechanics/.
Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model

R. James Cotton, J.D. Peiffer, Lucinda Williamson, and 2 more authors

In ECCV MoCha Workshop 2026, Aug 2026

arXiv:2608.29928 [cs.CV]

Abs DOI

State-of-the-art monocular body recovery methods predict mesh vertices and angles on the corresponding kinematic tree, but their outputs lack biomechanically defined joint angles that downstream applications like clinical and biomechanical analyses require. We extend an existing foundation model, SAM-3D-Body, with an additional biomechanical prediction head that, from a single RGB image, regresses the joint angles and scales of a biomechanical model. Training this model presents a challenge, as there are limited datasets of paired images and biomechanical fits. To overcome this, we supervise biomechanical outputs with in-loop optimized targets from a Levenberg-Marquardt solver performing inverse kinematics fits against markers from the mesh predictions. This allows distilling the biomechanical head from the mesh head, even from unlabeled images. To make this work with GPU-optimized biomechanical models in MuJoCo, the entire model was implemented in JAX using Equinox. We trained this distilled output head on the publicly released SAM-3D-Body dataset. We then validated this model on biomechanical fits to two publicly available marker-based datasets, MoVi and BioCV, as well as movements from a clinical cohort captured with multiview markerless motion capture. The resulting model outperforms existing models for direct regression of biomechanics from images while only slightly underperforming the state-of-the-art monocular biomechanics method that performs more costly inference-time optimization of entire trajectories.
Fiber Optic Sensing Glove for High Performance Dexterous Manipulation Capture

J.D. Peiffer, Taylor Niehues, Li Guan, and 2 more authors

In 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Aug 2026

arXiv:2608.24572 [cs.RO]

Abs DOI

Capturing hand pose during dexterous manipulation remains difficult: vision-based methods degrade under occlusion and challenging lighting, while sensorized gloves, though occlusion-free, are prone to drift and magnetic interference and rarely match motion-capture accuracy. We introduce a fiber optic sensing glove for full hand pose tracking that targets these failure modes, using multi-core shape-sensing fibers that capture each fiber’s full 3D shape rather than curvature alone. A novel pipeline registers each reconstructed fiber shape to a common hand reference frame, and a new inverse-kinematics solver reconstructs full hand pose at 60 Hz using curve constraints. Benchmarked on a 2-hour dataset of dexterous object manipulation tasks across 5 subjects, the glove achieves 7.2 mm mean fingertip position error against motion capture ground truth, reduced to 4.9 mm by a one-time factory calibration of the fiber routing hub that transfers across users and sessions. These capabilities enable high-fidelity data capture and bimanual virtual teleoperation — both essential to advancing the robotics field.
Differentiable Biomechanics for Markerless Motion Capture in Upper Limb Stroke Rehabilitation: A Comparison With Optical Motion Capture

Tim Unger, Arash Sal Moslehian, J.D. Peiffer, and 5 more authors

IEEE Transactions on Medical Robotics and Bionics, Feb 2026

Abs DOI

Marker-based Optical Motion Capture (OMC) paired with biomechanical modeling is currently considered the most precise and accurate method for measuring human movement kinematics. However, combining differentiable biomechanical modeling with Markerless Motion Capture (MMC) offers a promising approach to motion capture in clinical settings, requiring only minimal equipment, such as webcams, and minimal effort for data collection. This study compares key kinematic outcomes from biomechanically modeled MMC and OMC data in 15 individuals with stroke performing the drinking task, a functional task recommended for assessing upper limb movement quality. We observed a high level of agreement in kinematic trajectories between MMC and OMC, as indicated by high correlations (median r > 0.95 for the majority of kinematic trajectories) and median RMSE (root mean squared error) values ranging from 2°-5° for joint angles, 0.04 m/s for end-effector velocity, and 6 mm for trunk displacement. Trial-to-trial biases between OMC and MMC were consistent within participant sessions, with interquartile ranges of bias around 1-3° for joint angles, 0.01 m/s in end-effector velocity, and approximately 3 mm for trunk displacement. Our findings indicate that our MMC for arm tracking is approaching the accuracy of marker-based methods, supporting its potential for use in clinical settings. MMC could provide valuable insights into movement rehabilitation after stroke, potentially enhancing the effectiveness of rehabilitation strategies.
Monocular Markerless Motion Capture Enables Quantitative Assessment of Upper Extremity Reachable Workspace

Seth Donahue, J.D. Peiffer, R. Tyler Richardson, and 7 more authors

Sensors, Jan 2026

Abs DOI

This study validates a clinically accessible approach for quantifying the Upper Extremity Reachable Workspace (UERW) using monocular AI-driven Markerless Motion Capture (MMC). Objective validation of such techniques for clinically oriented tasks is essential to support their adoption in clinical motion analysis. Nine adults without impairments performed the standardized UERW task, reaching targets distributed across a virtual sphere centered on the torso and displayed via VR headset. Movements were simultaneously captured with a marker-based system and eight FLIR cameras; monocular analysis was applied to two videos representing frontal and offset camera configurations. Agreement was assessed by comparing the percentage workspace reached across six of eight workspace octants between the systems. The frontal camera demonstrated strong agreement with the marker-based reference (mean bias: 0.61±0.12% reachspace per octant), whereas the offset view underestimated workspace reached −5.66±0.45%. Depth-related errors in the frontal configuration were confined to posterior octants, whereas the offset view introduced inaccuracies in both contralateral and posterior octants. These findings support the feasibility of a frontal monocular camera for UERW assessment, particularly for anterior workspace evaluation. While posterior accuracy remains limited by depth estimation and anatomical occlusion errors, the overall results demonstrate clinical potential for practical, monocular-camera assessments.
Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements

Pouyan Firouzabadi, J.D. Peiffer, Kunal Shah, and 4 more authors

Jul 2026

arXiv:2607.02796 [cs.CV]

Abs DOI

Markerless motion capture (MMC) techniques have been widely beneficial in biomechanical analysis of human movement; however, application to complex motions of the hand lags other musculoskeletal systems. The primary goal of this study was to evaluate the performance of a biomechanical reconstruction method that implements a gradient-based optimization approach with a biomechanical model in the loop for tracking dexterous, unconstrained hand movements using MMC. Using a custom, 8-camera setup, we acquired 121 video recordings from 6 participants performing 11 different tasks that spanned 6 hand postures, 5 object manipulation tasks, and involved motion of the proximal upper limb joints. Performance of the proposed MMC pipeline was directly compared to a more commonly adopted two-stage reconstruction method that first triangulates 2D keypoints from computer vision pose estimation algorithms to 3D and then enforces biomechanical constraints by solving a constrained inverse kinematics problem. Relative performance was assessed qualitatively by visual inspection and quantitatively using a computer vision metric. Our method generated solutions for all 121 video recordings; the two-stage method did not converge for 15% of the recordings. Across the remaining videos, our method produced more biomechanically plausible hand kinematics than the two-stage method and was more robust to occlusion effects during tasks that involved objects. The relative robustness of the end-to-end method suggests that it is more effective in utilizing the available 2D digital keypoint information. Automatic and biomechanically meaningful tracking of hand kinematics during dexterous movements has the potential to support clinical evaluation, rehabilitation monitoring, and studies of human motor control.
Markerless Motion Capture for Biomechanical Whole-Body Kinematic Estimation in Infants

Divya Joshi, J.D. Peiffer, Colleen Peyton, and 1 more author

In 2026 48th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), May 2026

arXiv:2605.17120 [cs.CV]

Abs DOI

Early identification of motor impairment in infancy relies on expert visual assessment of spontaneous movement, motivating the development of automated, objective alternatives. One promising approach is using computer vision, which benefits from high quality pose estimation from video. In this study, we systematically evaluated three state-of-the-art pose estimation frameworks (MeTRAbs-ACAE, SAM 3D Body, and Sapiens) on 100 videos over 13 sessions of 8 infants recorded with a multi-view markerless motion capture system. We quantified keypoint detection accuracy using reprojection error, geometric consistency, and Procrustes-aligned 3D position error, and demonstrated proof-of-concept for fitting an inverse kinematic framework to infant data. While Sapiens achieved the lowest reprojection error and highest geometric consistency of the methods evaluated (22.8 pixels and 0.82, respectively), SAM 3D Body provided the most comprehensive 3D information for kinematic reconstruction with Procrustes-aligned position errors of 19 to 28 mm. We demonstrate in a case comparison example that biomechanical models fit to SAM 3D estimates distinguish representative movement patterns in infants related to motor development, as identified by a clinical expert. Together, these findings highlight both the promise and current limitations of 3D pose estimation for infant biomechanics and establish preliminary groundwork for scalable, video-based assessment of early motor development.
Synergy-based functional electrical stimulation for gait rehabilitation in chronic stroke: a pilot study

Jackson Taylor Levine, Xin Yu, Rebecca Muñoz, and 10 more authors

Frontiers in Rehabilitation Sciences, Aug 2026

Abs DOI

Chronic stroke gait disorders involve impaired motor coordination. While high-intensity gait training (HIGT) is supported by current clinical practice guidelines, and Functional Electrical Stimulation (FES) to tibialis anterior addresses foot drop, extending FES to multiple muscles may improve functional outcomes. Using a FES neural sleeve capable of recording muscle activity and joint angles, we tested preliminary efficacy and feasibility of a personalized multichannel FES (MFES) intervention based on the individual’s motor coordination impairment paired with HIGT. Fourteen individuals with chronic stroke were randomly assigned to either HIGT or MFES+HIGT for six weeks. Gait speed, endurance, gait biomechanics, and muscle synergies were assessed at baseline, midpoint, post-training, and one-month follow-up. Despite the small sample, only the MFES+HIGT group demonstrated significant endurance gains from baseline to post-training and follow-up, while both groups improved walking speed, impaired limb step length, and muscle synergy similarity to normative data. Feasibility was evaluated by measuring setup time (therapist putting device on participant and setting stimulation parameters) and collecting feedback from participants and four therapists. System setup time plateaued at 4.53 minutes by the ninth session. Both participants and therapists rated the intervention highly feasible, acceptable, and usable. Adherence was high in both groups, with only one dropout in the HIGT group due to preexisting medical conditions. This pilot study demonstrates the preliminary effectiveness of synergy-based MFES in improving walking endurance and confirms its feasibility for integration into chronic stroke gait rehabilitation, supporting the need for larger-scale trials to validate clinical efficacy and identify responders.

2024

Fusing Uncalibrated IMUs and Handheld Smartphone Video to Reconstruct Knee Kinematics

J.D. Peiffer, Kunal Shah, Shawana Anarwala, and 2 more authors

In 2024 10th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2024

ISSN: 2155-1782

Abs DOI

Video and wearable sensor data provide complementary information about human movement. Video provides a holistic understanding of the entire body in the world while wearable sensors, namely inertial measurement units (IMUs), provide high-resolution measurements of specific body segments. A robust method to fuse these modalities and obtain biomechanically accurate kinematics would have substantial utility for clinical assessment and monitoring. Although multiple video-sensor fusion methods exist, most assume that a time-intensive and often brittle sensor-body calibration process has already been completed. In this work, we employ an implicit function to combine handheld smartphone video and uncalibrated IMU data at their full temporal resolution. Our monocular, video-only, biomechanical reconstruction already performs well, with only 3.91 (1.55) degrees of mean adjusted angular error in knee kinematics across 60 recordings. Re-constructing from a fusion of video and IMU data reduces this error to 2.9 (1.27) degrees. We validate this method in a diverse group including individuals with no gait impairments, lower limb prosthesis users, and those with a history of stroke. We also show that IMU data allows accurate tracking through periods of visual occlusion, equivalent to video-only.
Hyperpolarized 129Xe MRI, 99mTc scintigraphy, and SPECT in lung ventilation imaging: a quantitative comparison

J.D. Peiffer, Talissa Altes, Iulian C. Ruset, and 6 more authors

Academic Radiology, 2024

DOI
Biomechanical Arm and Hand Tracking with Multiview Markerless Motion Capture

Pouyan Firouzabadi, Wendy Murray, Anton R Sobinov, and 4 more authors

In 2024 10th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), 2024

Abs DOI

Human arm and hand function is extremely complex with many degrees of freedom. It is also a common target for clinical interventions. However, precisely measuring upper extremity movement in both clinical and research settings is logistically challenging. We overcame this challenge through a novel approach to reconstructing arm biomechanics from markerless motion capture from multiple synchronized videos. Our approach directly opti-mizes the kinematics of an accurate biomechanical arm and hand that allows end-to-end minimization of the errors between the reconstructed movements and keypoints detected by computer vision. Key to this is an implicit function that maps from time to joint kinematics, which provides a learnable trajectory representation that can be differentiated through the biomechanical model, and supports GPU acceleration using MuJoCo-MJX. This optimization solves for the inverse kinematic solution consistent with the measured keypoints, consistent with biomechanical constraints, in addition to scaling the model while solving for the kinematics. We compare different hand keypoint detectors and find the best produces a fit with only several millimeters of reconstruction error. We also find that end-to-end optimization outperforms a two-stage fitting procedure, equivalent to more traditional biomechanical pipelines, where we first compute 3D marker trajectories and then perform inverse kinematics fitting in OpenSim. We anticipate this framework will reduce the barriers to biomechanical analysis of the arm and hand in both clinical and research settings.

2023

Enhanced selectivity of transcutaneous spinal cord stimulation by multielectrode configuration

Noah Bryson, Lorenzo Lombardi, Rachel Hawthorn, and 4 more authors

Journal of Neural Engineering, 2023

DOI
Self-Supervised Learning of Gait-Based Biomarkers

R. James Cotton, J.D. Peiffer, Kunal Shah, and 5 more authors

In Predictive Intelligence in Medicine, 2023

Abs DOI

Markerless motion capture (MMC) is revolutionizing gait analysis in clinical settings by making it more accessible, raising the question of how to extract the most clinically meaningful information from gait data. In multiple fields ranging from image processing to natural language processing, self-supervised learning (SSL) from large amounts of unannotated data produces very effective representations for downstream tasks. However, there has only been limited use of SSL to learn effective representations of gait and movement, and it has not been applied to gait analysis with MMC. One SSL objective that has not been applied to gait is contrastive learning, which finds representations that place similar samples closer together in the learned space. If the learned similarity metric captures clinically meaningful differences, this could produce a useful representation for many downstream clinical tasks. Contrastive learning can also be combined with causal masking to predict future timesteps, which is an appealing SSL objective given the dynamical nature of gait. We applied these techniques to gait analyses performed with MMC in a rehabilitation hospital from a diverse clinical population. We find that contrastive learning on unannotated gait data learns a representation that captures clinically meaningful information. We probe this learned representation using the framework of biomarkers and show it holds promise as both a diagnostic and response biomarker, by showing it can accurately classify diagnosis from gait and is responsive to inpatient therapy, respectively. We ultimately hope these learned representations will enable predictive and prognostic gait-based biomarkers that can facilitate precision rehabilitation through greater use of MMC to quantify movement in rehabilitation.
Markerless Motion Capture and Biomechanical Analysis Pipeline

R. James Cotton, Allison DeLillo, Anthony Cimorelli, and 5 more authors

2023

Abs DOI

Markerless motion capture using computer vision and human pose estimation (HPE) has the potential to expand access to precise movement analysis. This could greatly beneﬁt rehabilitation by enabling more accurate tracking of outcomes and providing more sensitive tools for research. There are numerous steps between obtaining videos to extracting accurate biomechanical results and limited research to guide many critical design decisions in these pipelines. In this work, we analyze several of these steps including the algorithm used to detect keypoints and the keypoint set, the approach to reconstructing trajectories for biomechanical inverse kinematics and optimizing the IK process. Several features we ﬁnd important are: 1) using a recent algorithm trained on many datasets that produces a dense set of biomechanically-motivated keypoints, 2) using an implicit representation to reconstruct smooth, anatomically constrained marker trajectories for IK, 3) iteratively optimizing the biomechanical model to match the dense markers, 4) appropriate regularization of the IK process. Our pipeline makes it easy to obtain accurate biomechanical estimates of movement in a rehabilitation hospital.
Optimizing Trajectories and Inverse Kinematics for Biomechanical Analysis of Markerless Motion Capture Data

R. James Cotton, Allison DeLillo, Anthony Cimorelli, and 5 more authors

In 2023 International Conference on Rehabilitation Robotics (ICORR), Singapore, Singapore, 2023

Abs DOI

Markerless motion capture using computer vision and human pose estimation (HPE) has the potential to expand access to precise movement analysis. This could greatly benefit rehabilitation by enabling more accurate tracking of outcomes and providing more sensitive tools for research. There are numerous steps between obtaining videos to extracting accurate biomechanical results and limited research to guide many critical design decisions in these pipelines. In this work, we analyze several of these steps including the algorithm used to detect keypoints and the keypoint set, the approach to reconstructing trajectories for biomechanical inverse kinematics and optimizing the IK process. Several features we find important are: 1) using a recent algorithm trained on many datasets that produces a dense set of biomechanically-motivated keypoints, 2) using an implicit representation to reconstruct smooth, anatomically constrained marker trajectories for IK, 3) iteratively optimizing the biomechanical model to match the dense markers, 4) appropriate regularization of the IK process. Our pipeline makes it easy to obtain accurate biomechanical estimates of movement in a rehabilitation hospital.