worker/docs/critical-check-cluster.md

# Dispatchless Critical-Check Cluster

## Status

`internal/workercluster` is a scaffold, not a production critical-check engine.
It currently provides Hashicorp Raft, bbolt log/stable storage, snapshots,
HTTP-based authenticated transport, bootstrap/join/membership operations, log
entry types, a small FSM, and operator status.

It does not yet execute `distributed_critical` checks, verify signed config,
evaluate observation/region/notification quorum, encrypt snapshots, deliver a
commit-backed outbox, or consume an external witness report.

The former hardcoded test-config endpoint, startup flag, environment switch,
and production helper have been removed. Hardcoded config application now
exists only as an unexported `_test.go` helper and is absent from production
builds.

## Non-Negotiable Separation

`distributed_critical` is a distinct path:

- no control-plane job dispatch;
- no PostgreSQL task row;
- no task lease or single worker owner;
- every eligible observer executes the adopted check deterministically;
- raw samples go to the control plane/TSDB outside Raft;
- classified observations, compact incidents, and outbox metadata enter Raft;
- normal checks and normal notifications continue unchanged.

The control plane signs configuration, receives replay, and acts as an external
witness. It is never a Raft voter.

## Topology

- Initial production clusters require exactly 3 voters. A future measured
  five-voter profile may be introduced separately; never use an even count.
- Voters remain within a bounded-latency topology. Remote regions use observer
  nodes unless measured RTT supports the configured election timeout.
- Nodes may be `voter`, `observer`, or `voter+observer`.
- Observer count affects observation policy, not Raft election quorum.
- A node without durable, real-fsync storage cannot be a voter.

Single-voter bootstrap is temporary. The cluster must not execute customer
critical checks until at least three voters are healthy and the signed
observer set is committed.

Membership and mTLS identity are assigned by the control plane as defined in
[public-endpoint-and-identity.md](public-endpoint-and-identity.md). Production
Raft traffic uses `/raft` on each worker's external HTTPS origin through an
HTTP/1.1 upgrade-capable reverse proxy. Static peers and shared Basic auth are
lab compatibility only.

## Raft State

The FSM contains only:

- monotonic config version and adopted critical-check definitions;
- versioned observer set;
- compact incident state and committed observations;
- bounded notification outbox metadata and idempotency tombstones;
- compact member diagnostics and partition/witness state;
- tenant isolation policy and applied-version index.

It never contains raw probe samples, response bodies, provider bodies,
plaintext credentials, credential envelopes, or normal task leases.

`raft-boltdb/v2` is only the local Raft `LogStore` and `StableStore`; it is not
the application model. Snapshots are produced by the FSM and must be versioned,
checksummed, and encrypted at rest before production use.

## Log Entries

Allowed application entry kinds:

- `config.adopt`
- `observer_set.update`
- `incident.observe`
- `incident.transition`
- `outbox.enqueue`
- `outbox.delivered`
- `outbox.ack`
- `partition.report`
- `membership.propose_add`
- `membership.demote`
- `membership.remove`
- `diagnostics.update`

No `task.lease.*`, retry, completion, or dead-letter entries belong in this
package.

## Signed Config Adoption

The control plane sends an Ed25519-signed canonical payload containing:

- cluster and tenant IDs;
- monotonic config version and expiry;
- critical checks and incident policies;
- proposed observer set, role/region map, and content hash;
- signing key ID and credential-envelope references.

Every node verifies signature, cluster identity, expiry, monotonic version,
observer-set hash, and supported schema. The leader proposes `config.adopt` and
`observer_set.update`. No observer executes the new config until both commits
are applied locally. Invalid config is rejected and reported through compact
diagnostics; there is no silent downgrade.

## Scheduling And Observation

For each adopted check, eligible observers derive the same interval boundary
from `epoch + n*interval`. Per-worker jitter is deterministic from worker ID,
check ID, observer-set version, and interval number. At each tick:

1. Run the underlying probe through `internal/checkexec` outside Raft.
2. Send raw metrics through the normal control-plane metrics path.
3. Classify to `ok`, `warn`, `down`, or `unknown` using signed policy.
4. Propose `incident.observe` to the leader with committed timestamp and
   observer/config versions.

Followers forward proposals or return a typed not-leader response containing
the current leader. They never silently drop observations.

## Four Quorums

- `raft_quorum`: majority of voters required to commit.
- `observation_quorum`: observers agreeing within the observation window.
- `region_quorum`: represented regions agreeing on state.
- `notification_quorum`: region agreement required for first customer alert.

All names remain distinct in code, metrics, configuration, and logs.

## Deterministic Incident FSM

`FSM.Apply` must perform no I/O, network calls, wall-clock reads, randomness,
or provider calls. It uses only committed payload values and prior state.

Incident lifecycle is `clear -> observing -> open -> resolving -> clear`.
Policy includes confirmation count, observation window, classification,
minimum incident dwell, transition-rate suppression, cooldown, and outbox retry
metadata. Apply enforces bounded state and deterministic idempotency keys.

The current FSM merely records the latest observation and accepts externally
constructed transitions. Replace this placeholder with deterministic policy
evaluation and tests before running real checks.

## Commit-Before-Notify

Strict order:

1. Probe and classify outside Raft.
2. Commit observation at `raft_quorum`.
3. FSM deterministically commits incident transition and outbox metadata.
4. After the committed outbox entry is visible, an executor calls the provider
   outside Raft.
5. Provider acknowledgement produces `outbox.delivered`; retry/failure produces
   bounded metadata updates.

Provider credentials stay in node-local memory/secure storage and are never
part of an entry or snapshot. Delivery uses stable channel idempotency keys to
survive leader failover.

## Partitions And Witness

Partition states are `steady`, `degraded`, `partitioned`, `healing`,
`split_brain_detected`, and `witness_only`. Loss of `raft_quorum` freezes
incident transitions and outbox creation. A cluster must never fabricate an
open or recovery while partitioned.

The external witness reports observed leader/term, reachable voters and
observers, split-brain flag/time, and report time. It can alert and inform state
but cannot commit or fabricate incidents.

## Ordered Implementation

1. [x] Remove production exposure of debug config application.
2. [ ] Complete versioned FSM types, command validation, and deterministic tests.
3. [ ] Adapt the rbackup CA pattern into short-lived SAN-bound mTLS identity,
   rotation/revocation, and safe one-claim cluster bootstrap.
4. [ ] Add signed config and observer-set adoption.
5. [ ] Add deterministic scheduler and checkexec bridge in shadow mode.
6. [ ] Implement observation aggregation, incident policy, and idempotency.
7. [ ] Implement encrypted snapshots and restore/migration tests.
8. [ ] Add metadata outbox executor and failover-safe delivery.
9. [ ] Add external witness, replay, metrics, and operational runbooks.
10. [ ] Run synthetic three-node HTTPS-proxy fault campaigns before any
    customer check; define a separate five-node profile before testing it.

## Release Gates

- Two independent bootstraps for one cluster are detected and rejected.
- Changes that leave fewer than three or an even number of voters are rejected.
- Leader failover completes within twice the configured election timeout.
- A minority partition commits zero incident transitions.
- Loss of observation quorum with intact Raft quorum sends no notification.
- Snapshot inspection finds no raw samples or credential material.
- Signature mutation and observer-set version mismatch prevent execution.
- Killing the leader between provider acknowledgement and metadata commit does
  not create duplicate customer-visible notification effects.
- Normal WebSocket tasks and notifications continue throughout cluster tests.