124 строки
5.7 KiB
Markdown
124 строки
5.7 KiB
Markdown
# Public Endpoint, Peer Identity, And Raft Transport
|
|
|
|
## Status
|
|
|
|
Accepted target architecture. Ordinary peer status checks already support an
|
|
external HTTPS worker URL. Raft currently uses a separate plaintext listener,
|
|
static peers, and shared Basic authentication; it does not yet meet this plan.
|
|
|
|
## One Worker, One Public URL
|
|
|
|
Every worker configures one absolute `PUBLIC_URL`, for example:
|
|
|
|
```text
|
|
https://worker-1.example.net
|
|
```
|
|
|
|
The worker proposes this URL during registration/configuration. The control
|
|
plane validates and canonicalizes it, persists the accepted value, and returns
|
|
it in signed init/config. The URL has scheme and authority only: no userinfo,
|
|
query, fragment, or ambiguous path. Public production URLs use HTTPS.
|
|
|
|
The control plane owns worker ID, account, region, cluster ID, membership,
|
|
voter/observer role, peer generation, and topology. A worker cannot gain trust
|
|
or Raft membership by advertising an endpoint.
|
|
|
|
## Routes On The Origin
|
|
|
|
- `/healthz`: process liveness;
|
|
- `/api/peer/status`: authenticated peer/control-plane reachability and Raft
|
|
summary;
|
|
- `/raft`: Hashicorp Raft HTTP/1.1 upgrade transport and membership requests;
|
|
- `/web/*` and `/web/api/*`: operator console.
|
|
|
|
TLS termination belongs to Traefik or nginx. The worker may keep internal HTTP
|
|
listeners on loopback/private addresses. `PUBLIC_URL` is advertisement, not a
|
|
request to bind the worker process or provision a public certificate.
|
|
|
|
For `/raft`, proxies must preserve path, query, Host, authorization/client
|
|
identity, `Connection`, and `Upgrade`; use HTTP/1.1, disable response buffering,
|
|
and allow long-lived upgraded connections. nginx requires explicit upgrade
|
|
headers. Tests must exercise this proxy path rather than only direct TCP.
|
|
|
|
## In-Memory Peer Database
|
|
|
|
Signed control-plane config supplies a monotonic topology generation and peers
|
|
scoped to one cluster/trust boundary. Each entry includes worker ID,
|
|
`PUBLIC_URL`, role, region, certificate identity, membership state, and expiry.
|
|
|
|
The worker replaces its in-memory peer database atomically when a newer valid
|
|
generation arrives. Removed or expired peers stop contributing immediately.
|
|
Periodic concurrent HTTP checks call each peer's `/api/peer/status`, validate
|
|
the TLS/client identity and response worker ID, and retain bounded observations.
|
|
Reachability never grants membership.
|
|
|
|
The response distinguishes process health, control-plane ping duration/result,
|
|
observation time, Raft role/term/leader, applied/commit indexes, and transport
|
|
errors. The worker console presents the same bounded per-peer state.
|
|
|
|
## Bind And Advertise Separation
|
|
|
|
Raft requires separate local bind and advertised addresses. The local listener
|
|
may be `127.0.0.1:37401`; the advertised endpoint derives from
|
|
`PUBLIC_URL + /raft`. Hashicorp Raft membership stores the advertised address,
|
|
never the loopback bind.
|
|
|
|
Outbound Raft transport performs TLS using the public hostname and worker
|
|
certificate. The current `rafthttp.NewDialTCP` and shared Basic-auth injection
|
|
must be replaced. Static `WORKER_CLUSTER_PEERS` remains a local lab fallback,
|
|
not production discovery.
|
|
|
|
## Certificate Authority
|
|
|
|
Adapt the proven CA shape from `/data/_backup/rbackup-server`, particularly its
|
|
explicit CA pool, required client-certificate verification, and certificate-to-
|
|
database identity lookup. Do not copy its tracked keys, ten-year certificates,
|
|
CN-only identity, inactive CRL, or server-generated private-key delivery.
|
|
|
|
The control plane owns a worker-cluster CA separate from public web
|
|
certificates. Requirements:
|
|
|
|
- worker-generated key and CSR;
|
|
- worker and cluster identity in a verified URI SAN;
|
|
- short-lived leaf certificates with automatic renewal;
|
|
- encrypted CA private key and documented backup/custody;
|
|
- issuance and revocation audit;
|
|
- emergency revocation plus short expiry as the normal revocation bound;
|
|
- rejection of expired, revoked, wrong-cluster, or SAN-mismatched peers.
|
|
|
|
One-time enrollment tokens authorize a CSR exactly once. They do not become
|
|
ongoing peer credentials. Browser traffic does not require a client certificate;
|
|
the reverse proxy applies mTLS policy to peer/Raft routes or forwards a verified
|
|
client identity through a tightly controlled internal boundary.
|
|
|
|
## Production Membership
|
|
|
|
Production critical-check clusters require exactly three voters initially.
|
|
Single-node bootstrap and two-node clusters are lab/bootstrap states and cannot
|
|
execute customer critical checks. Membership changes are leader-mediated and
|
|
control-plane-authorized. The control plane is an external witness, never a
|
|
voter.
|
|
|
|
## Implementation Order
|
|
|
|
1. Add `PUBLIC_URL` wire/config fields while accepting legacy `WORKER_URL` only
|
|
for a bounded migration.
|
|
2. Validate ownership/reachability and return accepted signed configuration.
|
|
3. Add scoped, versioned in-memory peer topology and concurrent health probes.
|
|
4. Extend status APIs/UI with networking, control-plane RTT, and Raft state.
|
|
5. Implement CA storage, CSR enrollment, renewal, revocation, and audit.
|
|
6. Split Raft bind/advertise addresses and add outbound TLS dialing.
|
|
7. Route `/raft` through the shared public origin and reverse-proxy contract.
|
|
8. Replace static production peers with control-plane-managed membership.
|
|
9. Run a three-voter TLS-proxy failover/partition campaign.
|
|
|
|
## Release Gates
|
|
|
|
- A peer cannot impersonate another worker or join another cluster.
|
|
- An invalid/replayed enrollment token issues no certificate.
|
|
- Revoked, expired, or SAN-mismatched certificates fail before Raft handling.
|
|
- Three workers join through external HTTPS origins and elect one leader.
|
|
- Killing one voter preserves quorum and commits through the proxy path.
|
|
- The UI reports each peer, transport state, control-plane RTT, role, term,
|
|
leader, and replication lag without exposing credentials.
|