Behavior Classification Inference Pipeline — 3.3× End-to-End
Bodit · 2026.02 · Led the rework · In production
- Python
- NumPy
- LightGBM
- MLflow
Reworked the production core package that extracts 903 features from 6-axis IMU raw signal, cutting end-to-end processing time 3.3×.
- IMU raw6-axis 25Hz
- Windowing24h → 8,640
- 903 feature extractionthe bottleneck
Vectorized the biggest bottleneck (cough features, 48.6% of runtime) 9.8× · third-party library → NumPy reimplementation
- ClassificationLightGBM
- Model loadingMLflow registry
Removed the local-file fallback so the served model is traceable
An input validation gate was added up front, and duplicate code was removed within the bounds of unchanged behavior.
Same time, same input
win/sec · end-to-end · 24h real-animal · 8,640 windows · 3.3×
Problem
A core package took 6-axis IMU raw signal from the sensors, cut it into windows, extracted 903 features and classified behavior. It was the bottleneck for the whole pipeline. Processing a single day of one animal took 65.8 seconds end to end, and every additional animal pushed the retraining and reprocessing cycle out further.
The catch: touching this code could change model output. If one feature’s value shifts even slightly, the trained classifier’s verdict wavers. Quietly trading correctness for speed was not an option.
What I did
I measured the bottleneck before assuming where it was. One feature set — cough-related features — was eating 48.6% of pipeline time (43.9 s), running about 10,000 peak-centered windows through per-row Python loops.
- Selectively vectorized that feature set: 61 of 63 features moved to NumPy batch operations, and 2 (entropy-based) stayed per-row. The goal was not to convert everything, only what paid off. That stage alone went from 43.9 s to 4.5 s (9.8×).
- Replaced six third-party (tsfresh) calls with a NumPy reimplementation. That stage got 37% faster and a dependency went away.
- On every swap, I checked the new batch functions numerically against the original per-row functions (within 1e-6, 36 unit tests).
- Moved model loading from a local-file fallback structure to a single registry path, making it traceable which model is actually being served.
- Added an input validation gate and removed code duplication within the bounds of unchanged behavior.
Result
Quietly trading correctness for speed was not an option. Every swap was checked numerically, and the values that did change were recorded, not hidden.
Measured on 24 hours of real-animal raw data — 2.16M rows / 8,640 windows. Full suite 208 passed / 0 failed. Every feature matched within 1e-6, but that tiny difference landed on a threshold and moved one cough statistic from 53 to 40 — recorded in the change log rather than hidden.
Measurement
- Subject
- 24 hours of real-animal IMU raw signal (6-axis, 25Hz)
- Sample
- 2.16M rows → 8,640 windows
- Baseline
- Same input before the rework · 65.8 s end-to-end (131 win/sec)
- Verification
- Batch functions checked numerically against the original per-row functions (atol 1e-6) · full suite 208 passed / 0 failed
Every number is end-to-end through the real call path (65.8 → 19.9 s). Summing per-stage timings gives a bigger 96 → 423 win/sec (4.4×), but that sum includes a stage outside the pipeline, so it is not the headline.