Datastores

Data categories (conceptual)

• Configuration state: devices/resources, mappings, environment metadata
• Versioned artifacts: snapshots/releases, change history, approvals
• Operational state: deployments, health signals, incident timelines

Design constraints

• Authoritative state + immutable history enables audits and rollback
• Stable identifiers make fleet status and timelines joinable
• Optimize hot queries early (fleet status, rollout views)

Architecture at a glance

• Define a stable artifact boundary (what you deploy) and a stable signal boundary (what you observe)
• Treat changes as versioned, testable, rollbackable units
• Use health + telemetry gates to scale safely

Typical workflow

• Define scope and success criteria (what should change, what must stay stable)
• Create or update a snapshot, then validate against a canary environment/site
• Deploy progressively with health/telemetry gates and explicit rollback criteria
• Confirm acceptance tests and operational dashboards before expanding

System boundary

Treat Datastores as a repeatable interface between engineering intent (design) and runtime reality (deployments + signals). Keep site-specific details configurable so the same design scales across sites.

Example artifact

topic: Datastores
plan: define -> snapshot -> canary -> expand
signals: health + telemetry + events tied to version
rollback: select known-good snapshot

Engineering outcomes

• Authoritative state + immutable history enables audits and rollback
• Stable identifiers make fleet status and timelines joinable
• Optimize hot queries early (fleet status, rollout views)

Quick acceptance checks

• Separate transactional config from large immutable artifacts
• Model audit trails as first-class (who/what/when) not an afterthought

Common failure modes

• Drift between desired and actual running configuration
• Changes without clear rollback criteria
• Insufficient monitoring for acceptance after rollout

Acceptance tests

• Verify the deployed snapshot/version matches intent (no drift)
• Run a canary validation: behavior, health, and telemetry align with expectations
• Verify rollback works and restores known-good behavior

Failure modes

• Inconsistent writes that create drift between UI and deployed state
• Slow queries on hot paths (e.g., fleet views) as the dataset grows
• Weak audit modeling that can’t answer “who changed what?”

Implementation checklist

• Separate transactional config from large immutable artifacts
• Model audit trails as first-class (who/what/when) not an afterthought
• Optimize hot queries (fleet views, status dashboards) as data grows
• Ensure consistency between UI state and deployed state via reconciliation

Rollout guidance

• Start with a canary site that matches real conditions
• Use health + telemetry gates; stop expansion on regressions
• Keep rollback to a known-good snapshot fast and rehearsed

Acceptance tests

• Verify the deployed snapshot/version matches intent (no drift)
• Run a canary validation: behavior, health, and telemetry align with expectations
• Verify rollback works and restores known-good behavior