worker/docs/critical-check-cluster.md

Dispatchless Critical-Check Cluster

Status

internal/workercluster is a scaffold, not a production critical-check engine. It currently provides Hashicorp Raft, bbolt log/stable storage, snapshots, HTTP-based authenticated transport, bootstrap/join/membership operations, log entry types, a small FSM, and operator status.

It does not yet execute distributed_critical checks, verify signed config, evaluate observation/region/notification quorum, encrypt snapshots, deliver a commit-backed outbox, or consume an external witness report.

The former hardcoded test-config endpoint, startup flag, environment switch, and production helper have been removed. Hardcoded config application now exists only as an unexported _test.go helper and is absent from production builds.

Non-Negotiable Separation

distributed_critical is a distinct path:

• no control-plane job dispatch;
• no PostgreSQL task row;
• no task lease or single worker owner;
• every eligible observer executes the adopted check deterministically;
• raw samples go to the control plane/TSDB outside Raft;
• classified observations, compact incidents, and outbox metadata enter Raft;
• normal checks and normal notifications continue unchanged.

The control plane signs configuration, receives replay, and acts as an external witness. It is never a Raft voter.

Topology

• Production clusters have 3 or 5 voters, never an even count.
• Voters remain within a bounded-latency topology. Remote regions use observer nodes unless measured RTT supports the configured election timeout.
• Nodes may be voter, observer, or voter+observer.
• Observer count affects observation policy, not Raft election quorum.
• A node without durable, real-fsync storage cannot be a voter.

Single-voter bootstrap is temporary. The cluster must not execute customer critical checks until at least three voters are healthy and the signed observer set is committed.

Raft State

The FSM contains only:

• monotonic config version and adopted critical-check definitions;
• versioned observer set;
• compact incident state and committed observations;
• bounded notification outbox metadata and idempotency tombstones;
• compact member diagnostics and partition/witness state;
• tenant isolation policy and applied-version index.

It never contains raw probe samples, response bodies, provider bodies, plaintext credentials, credential envelopes, or normal task leases.

raft-boltdb/v2 is only the local Raft LogStore and StableStore; it is not the application model. Snapshots are produced by the FSM and must be versioned, checksummed, and encrypted at rest before production use.

Log Entries

Allowed application entry kinds:

• config.adopt
• observer_set.update
• incident.observe
• incident.transition
• outbox.enqueue
• outbox.delivered
• outbox.ack
• partition.report
• membership.propose_add
• membership.demote
• membership.remove
• diagnostics.update

No task.lease.*, retry, completion, or dead-letter entries belong in this package.

Signed Config Adoption

The control plane sends an Ed25519-signed canonical payload containing:

• cluster and tenant IDs;
• monotonic config version and expiry;
• critical checks and incident policies;
• proposed observer set, role/region map, and content hash;
• signing key ID and credential-envelope references.

Every node verifies signature, cluster identity, expiry, monotonic version, observer-set hash, and supported schema. The leader proposes config.adopt and observer_set.update. No observer executes the new config until both commits are applied locally. Invalid config is rejected and reported through compact diagnostics; there is no silent downgrade.

Scheduling And Observation

For each adopted check, eligible observers derive the same interval boundary from epoch + n*interval. Per-worker jitter is deterministic from worker ID, check ID, observer-set version, and interval number. At each tick:

1. Run the underlying probe through internal/checkexec outside Raft.
2. Send raw metrics through the normal control-plane metrics path.
3. Classify to ok, warn, down, or unknown using signed policy.
4. Propose incident.observe to the leader with committed timestamp and observer/config versions.

Followers forward proposals or return a typed not-leader response containing the current leader. They never silently drop observations.

Four Quorums

• raft_quorum: majority of voters required to commit.
• observation_quorum: observers agreeing within the observation window.
• region_quorum: represented regions agreeing on state.
• notification_quorum: region agreement required for first customer alert.

All names remain distinct in code, metrics, configuration, and logs.

Deterministic Incident FSM

FSM.Apply must perform no I/O, network calls, wall-clock reads, randomness, or provider calls. It uses only committed payload values and prior state.

Incident lifecycle is clear -> observing -> open -> resolving -> clear. Policy includes confirmation count, observation window, classification, minimum incident dwell, transition-rate suppression, cooldown, and outbox retry metadata. Apply enforces bounded state and deterministic idempotency keys.

The current FSM merely records the latest observation and accepts externally constructed transitions. Replace this placeholder with deterministic policy evaluation and tests before running real checks.

Commit-Before-Notify

Strict order:

1. Probe and classify outside Raft.
2. Commit observation at raft_quorum.
3. FSM deterministically commits incident transition and outbox metadata.
4. After the committed outbox entry is visible, an executor calls the provider outside Raft.
5. Provider acknowledgement produces outbox.delivered; retry/failure produces bounded metadata updates.

Provider credentials stay in node-local memory/secure storage and are never part of an entry or snapshot. Delivery uses stable channel idempotency keys to survive leader failover.

Partitions And Witness

Partition states are steady, degraded, partitioned, healing, split_brain_detected, and witness_only. Loss of raft_quorum freezes incident transitions and outbox creation. A cluster must never fabricate an open or recovery while partitioned.

The external witness reports observed leader/term, reachable voters and observers, split-brain flag/time, and report time. It can alert and inform state but cannot commit or fabricate incidents.

Ordered Implementation

1. Remove production exposure of debug config application.
2. Complete versioned FSM types, command validation, and deterministic tests.
3. Add mTLS identity and safe one-claim cluster bootstrap.
4. Add signed config and observer-set adoption.
5. Add deterministic scheduler and checkexec bridge in shadow mode.
6. Implement observation aggregation, incident policy, and idempotency.
7. Implement encrypted snapshots and restore/migration tests.
8. Add metadata outbox executor and failover-safe delivery.
9. Add external witness, replay, metrics, and operational runbooks.
10. Run synthetic 3/5-node fault campaigns before any customer check.

Release Gates

• Two independent bootstraps for one cluster are detected and rejected.
• Changes that leave fewer than three or an even number of voters are rejected.
• Leader failover completes within twice the configured election timeout.
• A minority partition commits zero incident transitions.
• Loss of observation quorum with intact Raft quorum sends no notification.
• Snapshot inspection finds no raw samples or credential material.
• Signature mutation and observer-set version mismatch prevent execution.
• Killing the leader between provider acknowledgement and metadata commit does not create duplicate customer-visible notification effects.
• Normal WebSocket tasks and notifications continue throughout cluster tests.