Store-and-forward telemetry: design for the outage, not the demo

The network is a transport, not your historical record

Industrial links fail in ordinary ways: a customer replaces a router, a certificate expires, DNS breaks, a mobile uplink is weak, or maintenance isolates a cell. The machine should keep controlling locally, and the telemetry layer should know exactly which observations have not yet reached their durable destination. Treating an in-memory MQTT client queue as the record turns a reboot during an outage into permanent data loss.[1][4]

Give every sample enough identity to survive replay

A useful record carries machine ID, signal ID, value and quality, event timestamp, monotonic sequence, schema version, software version, and a stable record ID. Event time says when the machine observed the value; ingest time says when the platform received it. Mixing them makes a three-hour outage look like a sudden process event after reconnection.[6][5]

• Use a per-stream sequence number or durable unique record ID for deduplication.
• Mark clock quality and time-source changes; a timestamp is not automatically trustworthy.
• Version payload schemas so stored data remains decodable after software changes.
• Carry signal quality and stale/offline state rather than inventing numeric values.

Use a bounded local journal with explicit transactions

For compact machines, an embedded database or append-only log is often simpler and safer than a custom file format. A transaction can atomically accept a batch, while a write-ahead log separates durable writes from later checkpoint work. The important design choice is not SQLite versus another engine; it is having one component own ordering, acknowledgement, compaction, and crash recovery.[2][3]

Replay in order, but do not let history block the present

After reconnection, a controller may have hours of backlog while new observations continue. Uploading only old data delays the live view; uploading only new data can starve history forever. Use separate bounded lanes or weighted scheduling: reserve capacity for current health and alarms, then drain historical batches at a controlled rate.[1][5]

MQTT QoS 1 provides at-least-once delivery, which means duplicates are expected around reconnects and lost acknowledgements. Exactly-once business processing still requires the receiver to recognise record identity and make its write idempotent. QoS 2 is not a substitute for end-to-end data modelling.[1]

Decide what happens when storage fills

Calculate capacity from worst credible outage, sample rates, record overhead, write amplification, and safety margin. Then choose a policy per signal class. High-rate diagnostics may be downsampled; lifecycle counters may never be discarded; alarms may need a separate protected budget. "Keep everything" is not a policy when storage is finite.[4]

• Expose queue age, bytes used, oldest record, retry rate, and dropped-record count.
• Protect the control runtime from storage stalls with process and resource isolation.
• Make retention and downsampling rules part of the machine specification.
• Test power loss during write, checkpoint, upload, acknowledgement, and compaction.

A telemetry outage test worth running

1. Disconnect the uplink while the machine continues through representative states.
2. Reboot the edge service and, separately, power-cycle the controller during the outage.
3. Reconnect after the agreed maximum outage and keep generating live data.
4. Verify every retained record arrives once logically, with original event time and correct order per stream.
5. Confirm live status remains fresh during replay and the queue returns to zero.
6. Fill storage to each policy threshold and verify alarms, compaction, and controlled data loss.

A green dashboard after a five-minute Wi-Fi interruption is not enough. The acceptance result should quantify the longest retained outage, recovery bandwidth, replay time, duplicates received, records intentionally dropped, and any effect on the control cycle.