How historian compression distorts machine learning
The series in a historian is not the series the sensor saw. Compression settings change what features and models learn.
Process data is time-series data, and general-purpose databases handle it poorly. Historians are purpose-built for collecting, compressing, and serving millions of process values.
A temperature reading is not a fact — it is a sequence of facts: 98.2 °C at 10:00:00.0, 98.4 °C at 10:00:00.5, and so on. Every instrument in a plant produces such a stream. Relational databases, designed for rows you update in place, are a poor fit: a historian stores values that are written once, queried by time range, and never updated.
This is why data historians exist: specialized time-series databases built for the volume, write rate, and query patterns of process data.
Historians compress aggressively. Instead of storing every raw sample, most use swinging-door compression: a sample is stored only when the value moves outside a band around the last stored point. On a slowly changing temperature, millions of samples collapse to a handful of stored points — with the reconstruction error bounded by the band width.
Engineers must understand the trade-off: compression parameters define the accuracy of stored data. For regulatory or quality reporting, the compression band must be set tight enough to reconstruct values within the required tolerance. Never assume stored data is raw data.
Selection criteria: write throughput, query performance over long ranges, compression fidelity, connector coverage for your control systems, and the quality of its analytics API. Integration patterns:
The common denominator is the tag model: a consistent naming and metadata scheme across plants is what makes cross-site analysis possible at all.