What we learned building a historian that does not lose data
Deadbands, rollups, clock drift and the uncomfortable truth about fsync on industrial hardware.
A historian has one job that matters more than the others: when the power goes out mid-write, the samples you already acknowledged must still be there when it comes back.
That is harder than it sounds on the hardware historians actually run on.
Deadbands come first
The cheapest sample is the one you never store. Every tag has a configurable deadband, applied before the value reaches storage, so a temperature sitting in ±0.05 °C of noise does not generate a row every scan.
Two details bite people:
The deadband is in engineering units. A value of 0.1 is aggressive on a pH probe and meaningless on a flow meter reading 4,000 m³/h. Configure it per asset type, not globally.
Digital signals must not be deadbanded. A boolean has no meaningful interpolation — store every transition.
Rollups are what make long queries survivable
Raw samples answer "what happened at 14:32". They are a terrible way to answer "show me the last year". Rollups precompute the aggregates a long-range query needs so the chart does not have to scan a year of raw rows to draw four hundred pixels.
Clock drift is a data-integrity problem
An edge device that has been offline for two days often comes back with a clock that is wrong. If you trust it blindly, its backlog lands in the wrong place on the trend — sometimes in the future — and nothing about the chart tells the operator that happened.
Drift is corrected on ingest, against the server clock, before the samples are written. It is not the exciting part of a historian, and it is the part that stops a trend from quietly lying.
The uncomfortable part about fsync
A consumer-grade SSD in an industrial PC will frequently report that an fsync completed while the data is still in a volatile write cache. Under a clean shutdown you never notice. Under a power cut you lose the last few hundred milliseconds — silently, with no error anywhere.
There is no software fix for hardware that lies about durability. What we could do is be honest about it: document the failure mode, make the flush interval configurable so operators can trade throughput for durability knowingly, and default to a bound that a person can reason about.
Documenting that honestly has mattered more to the engineers we have shown it to than any benchmark number would have.
- #Performance
- #Historian
