A physical-therapy app that turns a phone — plus an optional IMU brace — into a closed-loop joint-angle tracker, giving clinicians richer movement data than a self-reported scale.
- Role
- Sensor fusion · ESP32 firmware · BLE
- Team
- 3 · with Mateo Taylor & Dustin Nguyen
- When
- Jan 2026 → ongoing
- Status
- Live on TestFlight (Feb 2026)
Overview
Doctors configure a patient's exercises; patients record themselves and get long-term recovery insights. 3D keypoints from RTMPose3D fuse with an optional ESP32 IMU brace over BLE through an Extended Kalman Filter that weights each camera update by detection confidence. Won the Healthcare track at Nexhacks @ CMU (1500+ participants) and went live on TestFlight in Feb 2026.
The brace
The hinged knee cuff splits into two shells that clamp the leg. Each shell carries its own MPU-6050 IMU; a XIAO ESP32-C6 reads both over I²C at 100 Hz and streams packetized orientation to the app over BLE every 3 measurements. This doesn't cause an issue for our kalman filter implementation since it's not in real-time, but rather gets processed after-the-fact. As the knee flexes, the hinge rotates the shells relative to each other — and the angle between the two IMUs is exactly the joint angle the EKF needs.
How it works
Camera + IMU time-sync over BLE
100 Hz IMU reads on the XIAO batch into 33.3 Hz BLE notifications (transmission every 3 measurements); a round-trip-time calibration (the app sends an ack, RTT/2 is added to each timestamp) aligns the IMU stream to the recorded video.
EKF for joint-angle fusion
State = [joint angle, IMU bias]. The IMU drives the angle at high rate; each CV estimate is a corrective update weighted by detection confidence — high-confidence frames pin the bias (CV angle − IMU angle).
Dual video pipeline
Frames stream to the Overshoot API for live activity classification while simultaneously recording locally; on session end the full clip runs through RTMPose3D for 3D keypoints used by the EKF.
Hardware-optional by design
The IMU brace sharpens accuracy but isn't required — the same EKF falls back to a constant-velocity model so the app still works phone-only.
Results
- 🏆 1st place — Healthcare track, Nexhacks @ CMU (1500+ participants).
- In-session activity classification plus per-session and cross-session joint-angle progress.
- Currently requires hefty cloud compute for skeleton-tracking.