worker/docs/public-endpoint-and-identity.md

# Public Endpoint, Peer Identity, And Raft Transport

## Status

Accepted target architecture. Ordinary peer status checks already support an
external HTTPS worker URL. Raft currently uses a separate plaintext listener,
static peers, and shared Basic authentication; it does not yet meet this plan.

Milestone 1 (worker side) is implemented: the worker configures `PUBLIC_URL`
as the canonical advertised origin with the legacy `WORKER_URL` accepted only
for a bounded migration, and validates the origin shape at startup and install
time. On the wire the worker consumes the accepted endpoint from
`wire.WorkerInit` (`public_url`, falling back to the legacy `url` field) and
rejects unusable values. Transmitting the proposed URL during registration is
the pending RSMon control-plane counterpart: it must populate `public_url` in
`WorkerInit` and accept `RegisterRequest.public_url`; until then the worker
reads whichever field the control plane sends.

## One Worker, One Public URL

Every worker configures one absolute `PUBLIC_URL`, for example:

```text
https://worker-1.example.net
```

The worker proposes this URL during registration/configuration. The control
plane validates and canonicalizes it, persists the accepted value, and returns
it in signed init/config. The URL has scheme and authority only: no userinfo,
query, fragment, or ambiguous path. Public production URLs use HTTPS.

The control plane owns worker ID, account, region, cluster ID, membership,
voter/observer role, peer generation, and topology. A worker cannot gain trust
or Raft membership by advertising an endpoint.

## Routes On The Origin

- `/healthz`: process liveness;
- `/api/peer/status`: authenticated peer/control-plane reachability and Raft
  summary;
- `/raft`: Hashicorp Raft HTTP/1.1 upgrade transport and membership requests;
- `/web/*` and `/web/api/*`: operator console.

TLS termination belongs to Traefik or nginx. The worker may keep internal HTTP
listeners on loopback/private addresses. `PUBLIC_URL` is advertisement, not a
request to bind the worker process or provision a public certificate.

For `/raft`, proxies must preserve path, query, Host, authorization/client
identity, `Connection`, and `Upgrade`; use HTTP/1.1, disable response buffering,
and allow long-lived upgraded connections. nginx requires explicit upgrade
headers. Tests must exercise this proxy path rather than only direct TCP.

## In-Memory Peer Database

Signed control-plane config supplies a monotonic topology generation and peers
scoped to one cluster/trust boundary. Each entry includes worker ID,
`PUBLIC_URL`, role, region, certificate identity, membership state, and expiry.

The worker replaces its in-memory peer database atomically when a newer valid
generation arrives. Removed or expired peers stop contributing immediately.
Periodic concurrent HTTP checks call each peer's `/api/peer/status`, validate
the TLS/client identity and response worker ID, and retain bounded observations.
Reachability never grants membership.

The response distinguishes process health, control-plane ping duration/result,
observation time, Raft role/term/leader, applied/commit indexes, and transport
errors. The worker console presents the same bounded per-peer state.

## Bind And Advertise Separation

Raft requires separate local bind and advertised addresses. The local listener
may be `127.0.0.1:37401`; the advertised endpoint derives from
`PUBLIC_URL + /raft`. Hashicorp Raft membership stores the advertised address,
never the loopback bind.

Outbound Raft transport performs TLS using the public hostname and worker
certificate. The current `rafthttp.NewDialTCP` and shared Basic-auth injection
must be replaced. Static `WORKER_CLUSTER_PEERS` remains a local lab fallback,
not production discovery.

## Certificate Authority

Adapt the proven CA shape from `/data/_backup/rbackup-server`, particularly its
explicit CA pool, required client-certificate verification, and certificate-to-
database identity lookup. Do not copy its tracked keys, ten-year certificates,
CN-only identity, inactive CRL, or server-generated private-key delivery.

The control plane owns a worker-cluster CA separate from public web
certificates. Requirements:

- worker-generated key and CSR;
- worker and cluster identity in a verified URI SAN;
- short-lived leaf certificates with automatic renewal;
- encrypted CA private key and documented backup/custody;
- issuance and revocation audit;
- emergency revocation plus short expiry as the normal revocation bound;
- rejection of expired, revoked, wrong-cluster, or SAN-mismatched peers.

One-time enrollment tokens authorize a CSR exactly once. They do not become
ongoing peer credentials. Browser traffic does not require a client certificate;
the reverse proxy applies mTLS policy to peer/Raft routes or forwards a verified
client identity through a tightly controlled internal boundary.

## Production Membership

Production critical-check clusters require exactly three voters initially.
Single-node bootstrap and two-node clusters are lab/bootstrap states and cannot
execute customer critical checks. Membership changes are leader-mediated and
control-plane-authorized. The control plane is an external witness, never a
voter.

## Implementation Order

1. Add `PUBLIC_URL` wire/config fields while accepting legacy `WORKER_URL` only
   for a bounded migration.

   Worker side implemented: `PUBLIC_URL` is canonical, `WORKER_URL` is a
   deprecated fallback with a startup warning, `internal/wire` carries
   `public_url` on `WorkerInit` (keeping `url` for compatibility), and
   startup/install enforce the strict origin shape for `PUBLIC_URL` while
   tolerating legacy `WORKER_URL` shapes. Control-plane counterpart: read
   `RegisterRequest.public_url` when worker-initiated registration is wired,
   populate `public_url` (not `url`) in `WorkerInit`, and persist the accepted
   origin. Until then the worker consumes whichever of `public_url`/`url` the
   control plane sends.
2. Validate ownership/reachability and return accepted signed configuration.
3. Add scoped, versioned in-memory peer topology and concurrent health probes.
4. Extend status APIs/UI with networking, control-plane RTT, and Raft state.
5. Implement CA storage, CSR enrollment, renewal, revocation, and audit.
6. Split Raft bind/advertise addresses and add outbound TLS dialing.
7. Route `/raft` through the shared public origin and reverse-proxy contract.
8. Replace static production peers with control-plane-managed membership.
9. Run a three-voter TLS-proxy failover/partition campaign.

## Release Gates

- A peer cannot impersonate another worker or join another cluster.
- An invalid/replayed enrollment token issues no certificate.
- Revoked, expired, or SAN-mismatched certificates fail before Raft handling.
- Three workers join through external HTTPS origins and elect one leader.
- Killing one voter preserves quorum and commits through the proxy path.
- The UI reports each peer, transport state, control-plane RTT, role, term,
  leader, and replication lag without exposing credentials.