Hardware usability is a machine learning problem long before it is an interface problem. A device can only adapt to its user if it can infer something true about them from indirect, noisy, physical signals. These four engagements were all versions of that question.
Four engagements, one question
Wearables
Hand gestures from wrist bio-signals, including ones users define. 90% accurate.
Health sensing
Stress detection that holds up outside the lab. 35% better.
AR / VR
Movement predicted a moment early, for HoloLens comfort. 80% accurate.
Gaming hardware
Grip position on a controller, inside the real sensor layout. 40% better.
Hand gestures, read from blood flow and motion
We built a model that detects hand gestures from bio-signals — IMU motion data and PPG, the optical blood-flow signal a wrist-worn device can already measure — for a next-generation wearable.
The hard requirement was not accuracy; it was that users could define their own gestures. That inverts the usual setup. A model trained on a fixed vocabulary has thousands of labelled examples per class. A user-defined gesture arrives with a handful, recorded once, by one person, on one wrist, with whatever fit and skin tone they happen to have. The model has to generalise from almost nothing, on a signal that varies substantially between people.
Stress detection that survives leaving the lab
Stress models work well in a laboratory and fall apart in daily life, for a reason that is physiological rather than statistical: climbing stairs, drinking coffee, a warm room and genuine stress all move the same signals in the same direction. A model trained on lab data learns "elevated" and calls it stress.
We designed a framework combining signal processing with ML classification and clustering, and — the part that mattered — brought task-context data in alongside the bio-signals, so the model could distinguish an elevated heart rate that has an obvious innocent explanation from one that does not.
Predicting movement before it happens, for HoloLens
Comfort in AR and VR is largely a latency problem, and latency cannot be solved by reacting faster — only by anticipating. If the system knows a moment in advance that the wearer is about to move, it buys back the milliseconds that otherwise register as discomfort.
We built a multi-modal model combining eye tracking, head movement and biomechanical signals to predict changes in body movement. Eye movement leads head movement, which leads body movement; the model exploits that ordering, using the earliest available signal to predict the latest.
Where the hands actually go on an Xbox controller
We developed a proximity-based hand-position classification model, using cluster analysis and XGBoost over sensor data combined with anthropometric measurements — because hand size and shape vary enormously, and a model that ignores that variation fits the average user and nobody else.
This one was constrained by physics rather than data. Sensor placement is decided by industrial design, electrical layout and manufacturing cost, not by what would be convenient for a classifier. We worked directly with industrial designers and electrical and biomechanical engineers, which meant designing the model around constraints we did not control.
What the four have in common
Every one of these is an inference problem where the ground truth is a person, and the measurement is indirect. You never observe a gesture, a stress state, an intended movement or a grip — you observe a voltage, an optical reflectance, an accelerometer trace, and you infer.
Which means the work that decides the outcome is nearly always upstream of the model: understanding what the sensor is physically measuring, what else produces the same reading, and what the hardware will permit you to measure in the first place. In four engagements across three very different products, no result turned on the choice of algorithm.
If you're facing something similar
If you're putting ML on a physical device, the binding constraint will come from the hardware — sensor placement, power budget, latency ceiling — and it will arrive after the modelling approach is already chosen if you let it. Bring those constraints in at the start and design the model inside them. Retrofitting a model to a sensor layout it was never designed for is the most common and most expensive way these projects go wrong.