ML
Isolation Forest for sensor anomalies: how it works and where it misleads
Read flagged rows in a sensor table by their isolation score, understand what contamination does in the model, and see where the method misleads you.
Prevent leakage in industrial time series
Why randomly splitting sensor rows can exaggerate real-world performance.
In short
- Adjacent readings often share the same conditions, so windows from one failure event must not land in both training and test sets.
- Scaling, imputation and feature selection must be fitted on training data only, because including the test mean in a scaler leaks future information.
- Choose the model, window and alert threshold on validation data, and keep the final test out of those decisions.
Rows are not independent
Adjacent readings often share the same conditions. Neighboring windows from one failure event can land in both training and test sets, exposing the event before evaluation. Many rows may still represent only a few independent failures. Report the number of rows, events and assets separately.
Consider both time and assets
For next-month prediction, train on the past and test on a later period. For deployment to new machines, hold out entire machines. Leave a gap when window lengths or prediction horizons overlap the boundary. A single split may not establish both new-asset and new-season generalization.
Preprocessing learns too
Fit scaling, imputation and feature selection on training data only. Including the test mean in a scaler leaks future information. Refit the preprocessing pipeline inside each validation fold. Check whether a maintenance outcome field was populated after the moment at which predictions would have been made.
Protect the final test
Choose the model, window and alert threshold using validation data. Keep the final test out of those decisions. If the result is poor, development can continue, but repeatedly using that test reduces its independence. Confirm the next release on a newly reserved period.
Frequently asked
How do I stop data leakage in time series for machine learning?
Do not split sensor rows randomly. When predicting next month, train on the past and test on a later period, and when targeting new machines, hold out entire machines. Leave a gap when window lengths or prediction horizons overlap the boundary, and keep the final test out of model decisions.
Why does preprocessing cause leakage in a machine learning pipeline?
Scaling, imputation and feature selection learn statistics from the data they see. If the test mean is included in a scaler, future information reaches training. Refit the preprocessing pipeline inside each validation fold, and check that no maintenance outcome field was filled in after the prediction time.
Sources and further reading
Pilot steps in this guide are proposed engineering workflows. Examples do not constitute measured field performance.
Questions on this topic
Keep reading.
Full library
ML
Multivariate process monitoring with Hotelling T² and SPE
Read correlated sensors through T² and SPE together rather than one statistic, interpret contribution plots, and learn where the method’s assumptions break.
ML
Without fault labels: unsupervised machine learning in industry
Learn what unlabelled sensor history can and cannot tell you, how to evaluate a detector without labels, and why operating modes need attention first.