DataOps
Prepare OPC UA data for AI
Turn a tag value into a usable observation with units, quality, timestamps and asset context.
How historian compression distorts machine learning
The series in a historian is not the series the sensor saw. Compression settings change what features and models learn.
In short
- A historian series is not the series the sensor saw, because a deadband or other lossy compression can change noise and peaks.
- A deadband can erase noise and produce an apparently flat stretch, which matters when the noise itself carries information.
- Forward-fill and linear interpolation are different assumptions, so the resampling method and its rationale should be recorded.
What lossy compression is
Many historian products do not store a value unless it leaves a band around the previously stored value (a deadband). Some drop points that a straight line can represent. This reduces storage and makes trend screens look good, but it leaves a different series from the raw one.
Which features are affected
The mean and last value are usually preserved. Standard deviation, derivatives, peaks and noise level change: a deadband erases noise and produces an apparently flat stretch. In predictive maintenance, where the vibration or current noise itself carries information, a compressed series may have removed it.
Resampling to regular intervals
Models usually expect an evenly spaced series. Forward-fill and linear interpolation are different assumptions when turning compressed records into a regular grid. Record the method and its rationale, and keep a separate flag for “actually measured” versus “derived”.
A practical checklist
Find out the compression setting for the tag you use. If possible, collect raw and compressed data side by side for a week and compare feature distributions. If the model will be trained on compressed data only, state that as a limitation. This guide is a suggested workflow; product-specific algorithm details should come from the vendor documentation.
Frequently asked
How does historian compression affect machine learning features?
Many historian products store a value only when it leaves a deadband around the previous stored value. The mean and last value are usually preserved, but standard deviation, derivatives, peaks and noise level can change. In predictive maintenance, where noise may carry information, a compressed series may have removed it.
What checklist should I follow before training on historian data?
Find the compression setting for the tag you use. If possible, collect raw and compressed data side by side for a week and compare feature distributions. If the model is trained only on compressed data, state that as a limitation. Use vendor documentation for product-specific algorithm details.
Sources and further reading
Pilot steps in this guide are proposed engineering workflows. Examples do not constitute measured field performance.
Questions on this topic
Keep reading.
Full library
DataOps
Check data quality before calculating OEE
Availability, performance and quality must use consistent production boundaries.
DataOps
Time-Series Data in Industry: Storage, Trends, and Forecasting
From historians to machine learning, time-series data powers industrial analytics. A practical guide to collection, quality, and the forecasting methods that actually work on plant data.