Public Endpoint, Peer Identity, And Raft Transport

Status

Accepted target architecture. Ordinary peer status checks already support an external HTTPS worker URL. Raft currently uses a separate plaintext listener, static peers, and shared Basic authentication; it does not yet meet this plan.

Milestone 1 (worker side) is implemented: the worker configures PUBLIC_URL as the canonical advertised origin with the legacy WORKER_URL accepted only for a bounded migration, and validates the origin shape at startup and install time. On the wire the worker consumes the accepted endpoint from wire.WorkerInit (public_url, falling back to the legacy url field) and rejects unusable values. Transmitting the proposed URL during registration is the pending RSMon control-plane counterpart: it must populate public_url in WorkerInit and accept RegisterRequest.public_url; until then the worker reads whichever field the control plane sends.

One Worker, One Public URL

Every worker configures one absolute PUBLIC_URL, for example:

https://worker-1.example.net

The worker proposes this URL during registration/configuration. The control plane validates and canonicalizes it, persists the accepted value, and returns it in signed init/config. The URL has scheme and authority only: no userinfo, query, fragment, or ambiguous path. Public production URLs use HTTPS.

The control plane owns worker ID, account, region, cluster ID, membership, voter/observer role, peer generation, and topology. A worker cannot gain trust or Raft membership by advertising an endpoint.

Routes On The Origin

/healthz: process liveness;
/api/peer/status: authenticated peer/control-plane reachability and Raft summary;
/raft: Hashicorp Raft HTTP/1.1 upgrade transport and membership requests;
/web/* and /web/api/*: operator console.

TLS termination belongs to Traefik or nginx. The worker may keep internal HTTP listeners on loopback/private addresses. PUBLIC_URL is advertisement, not a request to bind the worker process or provision a public certificate.

For /raft, proxies must preserve path, query, Host, authorization/client identity, Connection, and Upgrade; use HTTP/1.1, disable response buffering, and allow long-lived upgraded connections. nginx requires explicit upgrade headers. Tests must exercise this proxy path rather than only direct TCP.

In-Memory Peer Database

Signed control-plane config supplies a monotonic topology generation and peers scoped to one cluster/trust boundary. Each entry includes worker ID, PUBLIC_URL, role, region, certificate identity, membership state, and expiry.

The worker replaces its in-memory peer database atomically when a newer valid generation arrives. Removed or expired peers stop contributing immediately. Periodic concurrent HTTP checks call each peer's /api/peer/status, validate the TLS/client identity and response worker ID, and retain bounded observations. Reachability never grants membership.

The response distinguishes process health, control-plane ping duration/result, observation time, Raft role/term/leader, applied/commit indexes, and transport errors. The worker console presents the same bounded per-peer state.

Bind And Advertise Separation

Raft requires separate local bind and advertised addresses. The local listener may be 127.0.0.1:37401; the advertised endpoint derives from PUBLIC_URL + /raft. Hashicorp Raft membership stores the advertised address, never the loopback bind.

Outbound Raft transport performs TLS using the public hostname and worker certificate. The current rafthttp.NewDialTCP and shared Basic-auth injection must be replaced. Static WORKER_CLUSTER_PEERS remains a local lab fallback, not production discovery.

Certificate Authority

Adapt the proven CA shape from /data/_backup/rbackup-server, particularly its explicit CA pool, required client-certificate verification, and certificate-to- database identity lookup. Do not copy its tracked keys, ten-year certificates, CN-only identity, inactive CRL, or server-generated private-key delivery.

The control plane owns a worker-cluster CA separate from public web certificates. Requirements:

worker-generated key and CSR;
worker and cluster identity in a verified URI SAN;
short-lived leaf certificates with automatic renewal;
encrypted CA private key and documented backup/custody;
issuance and revocation audit;
emergency revocation plus short expiry as the normal revocation bound;
rejection of expired, revoked, wrong-cluster, or SAN-mismatched peers.

One-time enrollment tokens authorize a CSR exactly once. They do not become ongoing peer credentials. Browser traffic does not require a client certificate; the reverse proxy applies mTLS policy to peer/Raft routes or forwards a verified client identity through a tightly controlled internal boundary.

Production Membership

Production critical-check clusters require exactly three voters initially. Single-node bootstrap and two-node clusters are lab/bootstrap states and cannot execute customer critical checks. Membership changes are leader-mediated and control-plane-authorized. The control plane is an external witness, never a voter.

Implementation Order

Add PUBLIC_URL wire/config fields while accepting legacy WORKER_URL only for a bounded migration.

Worker side implemented: PUBLIC_URL is canonical, WORKER_URL is a deprecated fallback with a startup warning, internal/wire carries public_url on WorkerInit (keeping url for compatibility), and startup/install enforce the strict origin shape for PUBLIC_URL while tolerating legacy WORKER_URL shapes. Control-plane counterpart: read RegisterRequest.public_url when worker-initiated registration is wired, populate public_url (not url) in WorkerInit, and persist the accepted origin. Until then the worker consumes whichever of public_url/url the control plane sends.
Validate ownership/reachability and return accepted signed configuration.
Add scoped, versioned in-memory peer topology and concurrent health probes.
Extend status APIs/UI with networking, control-plane RTT, and Raft state.
Implement CA storage, CSR enrollment, renewal, revocation, and audit.
Split Raft bind/advertise addresses and add outbound TLS dialing.
Route /raft through the shared public origin and reverse-proxy contract.
Replace static production peers with control-plane-managed membership.
Run a three-voter TLS-proxy failover/partition campaign.

Release Gates

A peer cannot impersonate another worker or join another cluster.
An invalid/replayed enrollment token issues no certificate.
Revoked, expired, or SAN-mismatched certificates fail before Raft handling.
Three workers join through external HTTPS origins and elect one leader.
Killing one voter preserves quorum and commits through the proxy path.
The UI reports each peer, transport state, control-plane RTT, role, term, leader, and replication lag without exposing credentials.

6.9 KiB Исходник Ответственный История