From a1ccd50aaf4e4abda720345a51b73ce409e89aa0 Mon Sep 17 00:00:00 2001 From: Gleb Tv Date: Wed, 12 Aug 2026 18:40:43 +0300 Subject: [PATCH] docs: plan HTTPS peers and source install --- docs/README.md | 4 +- docs/architecture.md | 6 +- docs/control-plane-protocol.md | 13 ++- docs/critical-check-cluster.md | 15 +++- docs/implementation-roadmap.md | 13 ++- docs/install.md | 10 ++- docs/inventory.md | 9 +- docs/public-endpoint-and-identity.md | 123 ++++++++++++++++++++++++++ docs/source-installation.md | 89 +++++++++++++++++++ docs/source-plan-migration.md | 11 +-- docs/web-console-and-observability.md | 5 +- 11 files changed, 273 insertions(+), 25 deletions(-) create mode 100644 docs/public-endpoint-and-identity.md create mode 100644 docs/source-installation.md diff --git a/docs/README.md b/docs/README.md index 9907524..2c8f9dc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,6 +27,8 @@ disagree, update both in the same change or mark the discrepancy explicitly. | [private-workers.md](private-workers.md) | Customer worker isolation, credentials, onboarding | Partial | | [network-diagnostics.md](network-diagnostics.md) | Confirmation checks and dedicated diagnostic tasks | Partial | | [critical-check-cluster.md](critical-check-cluster.md) | Raft-backed dispatchless critical checks | Scaffold only | +| [public-endpoint-and-identity.md](public-endpoint-and-identity.md) | One HTTPS origin, peer status, CA/mTLS, managed Raft topology | Planned; peer HTTPS partial | +| [source-installation.md](source-installation.md) | Go SSH source installer and Docker/OpenSSH test matrix | Planned | | [implementation-roadmap.md](implementation-roadmap.md) | Ordered repository work packages and release gates | Active | | [source-plan-migration.md](source-plan-migration.md) | Source-to-target conversion ledger and resolved conflicts | Complete mapping | @@ -39,7 +41,7 @@ This repository owns: - check and notification execution; - local web console, local state, inventory collection, and host metrics; - worker-to-worker transport and Raft state; -- Docker, Compose, systemd, and release-image packaging. +- Docker, Compose, systemd, source installation, and release-image packaging. The RSMon control-plane repository owns: diff --git a/docs/architecture.md b/docs/architecture.md index 821db98..e8008d2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -78,8 +78,10 @@ provided by a validated control-plane task. Raft, and closes SQLite without exceeding the service stop timeout. - A web-console failure is reported and causes an intentional process policy; it must not silently leave a partially healthy process. -- Startup rejects malformed `WORKER_URL`, incomplete auth credentials, invalid - cluster settings, and unwritable data directories before accepting work. +- Startup currently rejects malformed `WORKER_URL`; the accepted target is + `PUBLIC_URL` with bounded compatibility migration. Startup also rejects + incomplete auth credentials, invalid cluster settings, and unwritable data + directories before accepting work. - `GET /healthz` reports process-local liveness. Control-plane reachability is a separate readiness/selfcheck signal and must not make a healthy container fail its local liveness probe. diff --git a/docs/control-plane-protocol.md b/docs/control-plane-protocol.md index 2d615e0..a2043f0 100644 --- a/docs/control-plane-protocol.md +++ b/docs/control-plane-protocol.md @@ -16,6 +16,9 @@ GET /worker?token= `/api/worker` and the HTTP jobs/results APIs remain compatibility paths. New workers use WebSocket task envelopes. +Public endpoint and cluster identity requirements are defined in +[public-endpoint-and-identity.md](public-endpoint-and-identity.md). + ## Frame Model Every frame is a `wire.WorkerMessage` with `kind` and one active content @@ -43,14 +46,20 @@ without execution or reporting. ## Initialization And Refresh -`wire.WorkerInit` supplies runtime values owned by the control plane: +The worker proposes `PUBLIC_URL` during registration. The control plane +validates and canonicalizes it; `wire.WorkerInit` returns the accepted endpoint +and supplies runtime values owned by the control plane: - worker ID, region, advertised URL, capabilities, and concurrency; - allowed notification methods and account IDs; - optional linked server ID for host metrics; - LLM endpoints; - scoped notification credentials and system contacts; -- peer workers for selfcheck/cluster-adjacent behavior. +- signed, cluster-scoped peer topology for selfcheck and Raft behavior. + +Worker ID, account, region, cluster, membership, role, topology generation, and +certificate identity are control-plane authority. Local environment or a peer +response cannot override them. Static peer environment remains lab-only. The worker clamps supplied concurrency to its local maximum. Credentials are replaced atomically in memory on refresh. Removed credentials must become diff --git a/docs/critical-check-cluster.md b/docs/critical-check-cluster.md index 7ab7cf1..cc36399 100644 --- a/docs/critical-check-cluster.md +++ b/docs/critical-check-cluster.md @@ -33,7 +33,8 @@ witness. It is never a Raft voter. ## Topology -- Production clusters have 3 or 5 voters, never an even count. +- Initial production clusters require exactly 3 voters. A future measured + five-voter profile may be introduced separately; never use an even count. - Voters remain within a bounded-latency topology. Remote regions use observer nodes unless measured RTT supports the configured election timeout. - Nodes may be `voter`, `observer`, or `voter+observer`. @@ -44,6 +45,12 @@ Single-voter bootstrap is temporary. The cluster must not execute customer critical checks until at least three voters are healthy and the signed observer set is committed. +Membership and mTLS identity are assigned by the control plane as defined in +[public-endpoint-and-identity.md](public-endpoint-and-identity.md). Production +Raft traffic uses `/raft` on each worker's external HTTPS origin through an +HTTP/1.1 upgrade-capable reverse proxy. Static peers and shared Basic auth are +lab compatibility only. + ## Raft State The FSM contains only: @@ -167,14 +174,16 @@ but cannot commit or fabricate incidents. 1. [x] Remove production exposure of debug config application. 2. [ ] Complete versioned FSM types, command validation, and deterministic tests. -3. [ ] Add mTLS identity and safe one-claim cluster bootstrap. +3. [ ] Adapt the rbackup CA pattern into short-lived SAN-bound mTLS identity, + rotation/revocation, and safe one-claim cluster bootstrap. 4. [ ] Add signed config and observer-set adoption. 5. [ ] Add deterministic scheduler and checkexec bridge in shadow mode. 6. [ ] Implement observation aggregation, incident policy, and idempotency. 7. [ ] Implement encrypted snapshots and restore/migration tests. 8. [ ] Add metadata outbox executor and failover-safe delivery. 9. [ ] Add external witness, replay, metrics, and operational runbooks. -10. [ ] Run synthetic 3/5-node fault campaigns before any customer check. +10. [ ] Run synthetic three-node HTTPS-proxy fault campaigns before any + customer check; define a separate five-node profile before testing it. ## Release Gates diff --git a/docs/implementation-roadmap.md b/docs/implementation-roadmap.md index 35ab619..4630678 100644 --- a/docs/implementation-roadmap.md +++ b/docs/implementation-roadmap.md @@ -14,6 +14,9 @@ Worker repository: - verify amd64 and arm64 image startup, Chromium availability, and version metadata; - add a package/install smoke test for Docker and systemd artifacts; +- add the Docker/OpenSSH source-install matrix for Alpine, Ubuntu, and Arch from + [source-installation.md](source-installation.md), using Go 1.26 and branch + `main`; - document immutable SHA and release tags as production defaults. Gate: a push publishes `sha-<12>` and `latest` manifests for both platforms, @@ -27,7 +30,8 @@ Worker repository: - wire local inventory and metrics collector lifecycle into web server start and shutdown; -- validate full HTTP config, including `WORKER_URL`, in main startup; +- validate full HTTP config, including the accepted `PUBLIC_URL`, in main + startup; - [x] reconnect in memory on token rotation without stopping the runner; - [x] resend bounded check/notification results after websocket reconnect; - define process policy when the web listener exits unexpectedly; @@ -70,7 +74,8 @@ Worker repository: - token handoff and revocation handling; - stale/expired signed-config behavior; - clear disabled/revoked UI state; -- optional mTLS client identity. +- mandatory production mTLS client identity and control-plane-owned topology as + defined in [public-endpoint-and-identity.md](public-endpoint-and-identity.md). Control-plane dependency: @@ -139,13 +144,13 @@ State: Raft scaffold only. Worker repository: -- secure bootstrap and mTLS identity; +- secure bootstrap, CA lifecycle, mTLS identity, and one-origin HTTPS transport; - signed config adoption and observer-set versioning; - deterministic scheduler and check executor bridge; - deterministic incident/quorum FSM; - encrypted snapshots and restore; - metadata outbox, witness, replay, and metrics; -- 3/5-node fault campaigns. +- three-node external-HTTPS fault campaigns. Control-plane dependency: diff --git a/docs/install.md b/docs/install.md index 8ca8a12..3eccbe3 100644 --- a/docs/install.md +++ b/docs/install.md @@ -5,7 +5,8 @@ Docker image) into a running, enabled systemd service on a Linux host. It is the supported way to deploy the worker: it writes the configuration, the systemd unit, and the data directory, then starts the service. -> A shell installer that downloads a pre-built binary is planned. Today +> A Go SSH source installer is planned in +> [source-installation.md](source-installation.md). Today > `install` copies the binary you invoke it from (or pulls the `--image` > digest), so build first with `make build` and run the resulting > `./bin/rsmon-worker`. @@ -53,7 +54,7 @@ values automatically. To override a value, pass the matching flag. | `RSMON_TOKEN` | yes | none | Worker bearer token. | | `WORKER_HOST` | no | `127.0.0.1` | Operator-console bind address. | | `WORKER_PORT` | no | `27401` (primary) | Operator-console port. **Required** for named instances. | -| `WORKER_URL` | no | none | Public URL advertised to the control plane. | +| `PUBLIC_URL` | no | none | Proposed public HTTPS origin; accepted by the control plane. | | `WORKER_LOGIN` | no | `admin` (generated) | Operator-console basic-auth login. | | `WORKER_PASSWORD` | no | generated | Operator-console basic-auth password. | | `WORKER_COMPOSE_ENABLED` | no | feature default (on) | Enable Docker Compose discovery/management. | @@ -66,6 +67,11 @@ writes it to the env file, and prints it once. Record it; the operator console requires it for both the browser login and the `/web/api/*` HTTP basic-auth endpoints. +`PUBLIC_URL` does not bind a listener or terminate TLS. It advertises the one +external origin used for the console, authenticated peer status, and planned +Raft `/raft` transport. See +[public-endpoint-and-identity.md](public-endpoint-and-identity.md). + Values must be systemd-safe: no whitespace, quotes, backslashes, or `$` interpolation inside a value. This keeps the file unambiguous across systemd `EnvironmentFile` and `docker --env-file`. diff --git a/docs/inventory.md b/docs/inventory.md index 485f801..6c4bb60 100644 --- a/docs/inventory.md +++ b/docs/inventory.md @@ -8,7 +8,7 @@ plane validates identity, applies account scope, reconciles lifecycle, and persists the inventory projection. The control-plane inventory entities remain `Server`, `ServerIp`, `Site`, -`Deployment`, `Domain`, `Repo`, and `site_repos`. Their API and rstuff stream +`Deployment`, `Domain`, `Repo`, and `site_repos`. Their API and legacy stream projection are control-plane concerns. This repository owns only local discovery and worker-originated reports. @@ -130,7 +130,7 @@ successful section can mark previously observed entities missing. - Do not guess a primary address; report interface and route metadata so the control plane can apply policy. -## Relationship To deploymentd And rstuff +## Relationship To deploymentd And RSLogin Worker reports and deploymentd ingest are complementary producers of the same control-plane projection. During migration: @@ -140,8 +140,9 @@ control-plane projection. During migration: - every row records source and source report ID; - reconciliation occurs per source and section to prevent one producer from deleting another producer's observations; -- the rstuff `rsmon.inventory.v1` stream remains a control-plane output/input, - not a worker protocol. +- the former rstuff stream direction is superseded; RSMon owns inventory and + RSLogin supplies trusted project and SSH-access input through a separate + control-plane integration, never a worker protocol. ## Delivery Phases diff --git a/docs/public-endpoint-and-identity.md b/docs/public-endpoint-and-identity.md new file mode 100644 index 0000000..6f9e532 --- /dev/null +++ b/docs/public-endpoint-and-identity.md @@ -0,0 +1,123 @@ +# Public Endpoint, Peer Identity, And Raft Transport + +## Status + +Accepted target architecture. Ordinary peer status checks already support an +external HTTPS worker URL. Raft currently uses a separate plaintext listener, +static peers, and shared Basic authentication; it does not yet meet this plan. + +## One Worker, One Public URL + +Every worker configures one absolute `PUBLIC_URL`, for example: + +```text +https://worker-1.example.net +``` + +The worker proposes this URL during registration/configuration. The control +plane validates and canonicalizes it, persists the accepted value, and returns +it in signed init/config. The URL has scheme and authority only: no userinfo, +query, fragment, or ambiguous path. Public production URLs use HTTPS. + +The control plane owns worker ID, account, region, cluster ID, membership, +voter/observer role, peer generation, and topology. A worker cannot gain trust +or Raft membership by advertising an endpoint. + +## Routes On The Origin + +- `/healthz`: process liveness; +- `/api/peer/status`: authenticated peer/control-plane reachability and Raft + summary; +- `/raft`: Hashicorp Raft HTTP/1.1 upgrade transport and membership requests; +- `/web/*` and `/web/api/*`: operator console. + +TLS termination belongs to Traefik or nginx. The worker may keep internal HTTP +listeners on loopback/private addresses. `PUBLIC_URL` is advertisement, not a +request to bind the worker process or provision a public certificate. + +For `/raft`, proxies must preserve path, query, Host, authorization/client +identity, `Connection`, and `Upgrade`; use HTTP/1.1, disable response buffering, +and allow long-lived upgraded connections. nginx requires explicit upgrade +headers. Tests must exercise this proxy path rather than only direct TCP. + +## In-Memory Peer Database + +Signed control-plane config supplies a monotonic topology generation and peers +scoped to one cluster/trust boundary. Each entry includes worker ID, +`PUBLIC_URL`, role, region, certificate identity, membership state, and expiry. + +The worker replaces its in-memory peer database atomically when a newer valid +generation arrives. Removed or expired peers stop contributing immediately. +Periodic concurrent HTTP checks call each peer's `/api/peer/status`, validate +the TLS/client identity and response worker ID, and retain bounded observations. +Reachability never grants membership. + +The response distinguishes process health, control-plane ping duration/result, +observation time, Raft role/term/leader, applied/commit indexes, and transport +errors. The worker console presents the same bounded per-peer state. + +## Bind And Advertise Separation + +Raft requires separate local bind and advertised addresses. The local listener +may be `127.0.0.1:37401`; the advertised endpoint derives from +`PUBLIC_URL + /raft`. Hashicorp Raft membership stores the advertised address, +never the loopback bind. + +Outbound Raft transport performs TLS using the public hostname and worker +certificate. The current `rafthttp.NewDialTCP` and shared Basic-auth injection +must be replaced. Static `WORKER_CLUSTER_PEERS` remains a local lab fallback, +not production discovery. + +## Certificate Authority + +Adapt the proven CA shape from `/data/_backup/rbackup-server`, particularly its +explicit CA pool, required client-certificate verification, and certificate-to- +database identity lookup. Do not copy its tracked keys, ten-year certificates, +CN-only identity, inactive CRL, or server-generated private-key delivery. + +The control plane owns a worker-cluster CA separate from public web +certificates. Requirements: + +- worker-generated key and CSR; +- worker and cluster identity in a verified URI SAN; +- short-lived leaf certificates with automatic renewal; +- encrypted CA private key and documented backup/custody; +- issuance and revocation audit; +- emergency revocation plus short expiry as the normal revocation bound; +- rejection of expired, revoked, wrong-cluster, or SAN-mismatched peers. + +One-time enrollment tokens authorize a CSR exactly once. They do not become +ongoing peer credentials. Browser traffic does not require a client certificate; +the reverse proxy applies mTLS policy to peer/Raft routes or forwards a verified +client identity through a tightly controlled internal boundary. + +## Production Membership + +Production critical-check clusters require exactly three voters initially. +Single-node bootstrap and two-node clusters are lab/bootstrap states and cannot +execute customer critical checks. Membership changes are leader-mediated and +control-plane-authorized. The control plane is an external witness, never a +voter. + +## Implementation Order + +1. Add `PUBLIC_URL` wire/config fields while accepting legacy `WORKER_URL` only + for a bounded migration. +2. Validate ownership/reachability and return accepted signed configuration. +3. Add scoped, versioned in-memory peer topology and concurrent health probes. +4. Extend status APIs/UI with networking, control-plane RTT, and Raft state. +5. Implement CA storage, CSR enrollment, renewal, revocation, and audit. +6. Split Raft bind/advertise addresses and add outbound TLS dialing. +7. Route `/raft` through the shared public origin and reverse-proxy contract. +8. Replace static production peers with control-plane-managed membership. +9. Run a three-voter TLS-proxy failover/partition campaign. + +## Release Gates + +- A peer cannot impersonate another worker or join another cluster. +- An invalid/replayed enrollment token issues no certificate. +- Revoked, expired, or SAN-mismatched certificates fail before Raft handling. +- Three workers join through external HTTPS origins and elect one leader. +- Killing one voter preserves quorum and commits through the proxy path. +- The UI reports each peer, transport state, control-plane RTT, role, term, + leader, and replication lag without exposing credentials. diff --git a/docs/source-installation.md b/docs/source-installation.md new file mode 100644 index 0000000..9d2c3dd --- /dev/null +++ b/docs/source-installation.md @@ -0,0 +1,89 @@ +# SSH Source Installation Plan + +## Status + +Planned. The current Go installer can upload a binary or deploy an immutable +Docker image over SSH. It does not yet install build prerequisites, download Go, +clone the public repository, or build remotely. Existing tests are unit tests; +there is no live OpenSSH-container installation test. + +## Initial Platform Scope + +The first source installer supports Linux and is validated on Alpine, Ubuntu, +and Arch Linux containers. CentOS-family support follows after its package and +service differences are implemented. Windows and macOS remain later platform +work despite the worker being written in Go. + +Use `reg.rsxx.ru` image mirrors where available. Tests must not depend on Docker +Hub when a local mirror exists. + +## Source Install Flow + +The Go CLI connects through `golang.org/x/crypto/ssh` using existing key, +passphrase, password, sudo-password, known-hosts, and pinned-fingerprint support. +It then: + +1. detects supported OS, architecture, package manager, init system, and + privilege path; +2. installs only required packages (`git`, CA certificates, download/archive + tools); it does not install `build-essential` or a C compiler unless a + detected dependency requires CGO; +3. downloads the pinned Go 1.26 toolchain for the detected architecture and + verifies the published SHA-256; +4. clones `https://rocketgit.ru/rsmon/worker.git` or updates an existing clone; +5. checks out branch `main` and records the resolved commit; +6. builds a reproducible worker binary with the repository build flags; +7. atomically installs the binary, validated environment, data directory, and + service definition; +8. starts the service and verifies process status and `/healthz`. + +Repository, branch, Go version, checksum source, build directory, and Go module +proxy may be configurable, but production output records their resolved values. +The default repository is publicly readable and requires no source credential. + +## Docker OpenSSH Test Harness + +Adapt the real-network pattern from `/data/_swap/sshkeymanager`: start an +OpenSSH container, wait for SSH readiness, connect with the Go installer, and +tear the environment down reliably. Do not mock SSH command execution in the +acceptance test. + +Provide images/fixtures for: + +- Ubuntu with apt and systemd-compatible service testing where practical; +- Alpine with apk/OpenRC or a clearly separated no-service build/install gate; +- Arch with pacman and its service behavior. + +Each clean target begins without Go or the worker source. The test asserts +package installation, verified Go version, clone branch/resolved commit, build, +atomic config permissions, running service where supported, and HTTP liveness. + +## Idempotency And Security + +- A second run updates/fetches safely and leaves one active service. +- Wrong host fingerprints fail before remote mutation. +- Tokens/passwords come from files or stdin-safe channels and never appear in + command arguments, logs, source checkout, or shell history. +- Remote temporary files are removed on success and failure. +- Failed builds do not replace a working binary or service definition. +- Package-manager and download failures return bounded actionable errors. +- The installer verifies Go tarball checksum before extraction. + +## Implementation Work Packages + +1. Add reusable Docker/OpenSSH harness and distro fixtures. +2. Add pure distro/toolchain/source-install script planning and unit tests. +3. Execute source installation through the existing SSH transport. +4. Add atomic build/install, idempotency, and failure rollback. +5. Add Alpine, Ubuntu, and Arch network E2E tests to CI. +6. Add CentOS-family support. +7. Plan native Windows service and macOS launchd installers separately. + +## Acceptance Gates + +- All three initial Linux images install from a clean state through OpenSSH. +- The built worker reports the expected version/commit and serves `/healthz`. +- Re-running the installer succeeds without duplicate services or leaked files. +- Host-key, checksum, clone, build, and service-start failure tests preserve the + previous installation. +- CI uses approved registry mirrors and cleans every test container/network. diff --git a/docs/source-plan-migration.md b/docs/source-plan-migration.md index 718d960..a5b5557 100644 --- a/docs/source-plan-migration.md +++ b/docs/source-plan-migration.md @@ -39,10 +39,10 @@ Conversion date: 2026-07-13. | --- | --- | --- | | `docs/inventory-management.md` | `inventory.md` | Control-plane projection retained as dependency; worker owns collection only | | `docs/plans/inventory-management.md` | `inventory.md` | deploymentd and worker reports defined as source-aware complementary producers | -| `docs/parity/rstuff-inventory.md` | `inventory.md` | rstuff enums/stream remain control-plane contracts, not worker protocol | +| `docs/parity/rstuff-inventory.md` | `inventory.md` | Historical only; RSMon owns inventory and RSLogin supplies project/access input | | `docs/parity/deploymentd.md` | `inventory.md` | deploymentd parity becomes fixture normalization and comparison gate | | `docs/deploymentd-replay-rollout.md` | `inventory.md` | HTTP nonce rollout remains deploymentd-specific; worker uses authenticated typed reports | -| `docs/integrations/rstuff-inventory-stream.md` | `inventory.md` | Valkey stream remains between control plane and rstuff, never worker transport | +| `docs/integrations/rstuff-inventory-stream.md` | `inventory.md` | Superseded; no rstuff service is planned and Valkey is not the target inventory authority | | `docs/plans/servers-and-hardware-metrics.md` | `web-console-and-observability.md`, `inventory.md` | Implemented server metric report retained; local hardware extensions staged | | `docs/server-observability.md` | `web-console-and-observability.md`, `control-plane-protocol.md` | Current wire fields, ownership validation, and TSDB boundary retained | | `docs/parity/capture.md` | `web-console-and-observability.md`, `implementation-roadmap.md` | Worker-host coverage gaps converted into staged collector work | @@ -101,9 +101,10 @@ obsolete. Collector lifecycle remains a real implementation gap. ### Inventory deploymentd HTTP ingest is currently implemented on the control plane. Worker -inventory is local/partial and has no upstream report yet. Future worker and -deploymentd reports are complementary source-scoped inputs; neither may erase -the other's observations. +inventory is local/partial and has no upstream report yet. RSMon is the target +inventory authority; RSLogin supplies trusted project and SSH-access input. +Future worker and deploymentd reports are complementary source-scoped inputs; +neither may erase the other's observations. ### Private Workers And Public Work diff --git a/docs/web-console-and-observability.md b/docs/web-console-and-observability.md index 7cbb110..fc2a237 100644 --- a/docs/web-console-and-observability.md +++ b/docs/web-console-and-observability.md @@ -27,8 +27,9 @@ absolute lifetime. State-changing browser requests require CSRF validation. Authenticated responses use no-store and restrictive security headers. OAuth/device authorization is not implemented. Public exposure requires a TLS -reverse proxy and network restrictions. Do not claim that `WORKER_URL` enables -TLS; it only advertises the externally reachable URL. +reverse proxy and network restrictions. The accepted target name is +`PUBLIC_URL`; `WORKER_URL` is current legacy configuration during migration. +Neither setting enables TLS; it only advertises the externally reachable URL. ## Known Runtime Gap