Footsteps are a signal. We learned to read it.
Every step is a short burst of force that rings through the floor. Happy Steps separates the person from the floor, and turns what is left into identity, gait and change over time.
足音は信号です。床の響きから「人」を取り出します。
Source × filter, like a voice.
In speech, sound is a source (the vocal cords) shaped by a filter (the vocal tract). A footstep works the same way. The foot strike is the source. The floor is the filter.
The same person sounds different on wood and on concrete, just as one speaker sounds different through two microphones. The slow envelope of the floor response follows the force of the foot, so separating person from floor becomes a blind deconvolution problem. Solving it is what makes the method work in real homes.
Five steps from vibration to meaning.
1 · Footstep isolation
An unsupervised Gaussian mixture, fitted with EM, separates footstep events from background vibration. No labels needed.
2 · Time–frequency view
A continuous wavelet transform with a complex Morlet wavelet shows each step as a picture: sharp in time at high pitch, fine in frequency at low pitch.
3 · DeepStep identity
A residual network with attention, trained with a contrastive open-set loss, says who walked, or that this is someone new.
4 · Gait health
A multi-task ResNet-18 flags unusual gait. Step timing gives pace, steadiness and rhythm balance.
5 · Cognitive load (research)
A dual-task walking study shows the mind’s load changes the footstep, something that usually needs EEG or questionnaires.
+ Gemini on top
The outputs become small events. Gemini reasons over them across days, and talks to people.
Measured, not promised.
From the VIBeID dataset: 100+ participants, 96+ hours of footsteps, recorded on wood, carpet, cement and outdoor surfaces at different walking speeds.
| Task | Result | What it means |
|---|---|---|
| Person identification (100+ people) | 92.68% Rank-1 | About 1 wrong in 14 |
| Spotting an unknown person (open-set) | 93.83% AUC | Visitors are recognised as “not the resident” |
| Gait-abnormality detection | 93.67% | About 1 missed in 16 |
| Cognitive load from walking | 68.49% | Better than 2-to-1 odds over chance; early research |
| Fusion of tasks | 89.10% | Joint model across outputs |
| On-device cost | 22 MB · 0.78 s | Runs on a Raspberry Pi, no cloud needed |
All numbers are from controlled lab studies in the author’s doctoral thesis. Real homes are harder; see limits below.
Peer-reviewed foundations.
The methods behind Happy Steps are published in five journals and five conferences, with four patents.
- IEEE Transactions on Mobile Computing
- Scientific Data (Nature portfolio) · the VIBeID dataset
- IEEE Signal Processing Letters
- IEEE Sensors Journal
- IEEE Transactions on Instrumentation and Measurement
Honest limits
- Each new floor needs a short setup walk to learn how it rings.
- Two people walking at once is harder; we tag those walks as mixed.
- Lab results will drop in busy homes with washing machines and pets. A home pilot is the next step.
- It is a care helper, not a medical device. It describes change; it never diagnoses.
What we are building next.
Synthetic footsteps
- Generate floor signals from video of walking
- Like text-to-speech, but for footsteps
- New floors without new recordings
Smaller, local reasoning
- TinyML step models on microcontrollers
- Explore on-device Gemini Nano for offline notes
- Lower cost per room
Longer memory
- Months of baseline, seasonal patterns
- Early, gentle trend reports for families
- Studies with care professionals
See a real footstep become a kind word.
The live demo runs real floor recordings through the full pipeline and calls Gemini in your browser.