diff --git a/README.md b/README.md index 0e8779f..a9c369a 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,12 @@ Published images use these tags: - `sha-<12-character-commit>` for every push; - `latest` for `master`; -- the exact `v*` tag for releases. +- the `v*` release ref, with Docker-invalid characters replaced by `-`. + +The Gitea workflow reads `HARBOR_REGISTRY`, `HARBOR_USER`, and +`HARBOR_PASSWORD`. `HARBOR_REGISTRY` may be a host such as `reg.rsxx.ru` or an +HTTP(S) URL; the workflow strips the scheme and trailing slash before composing +Docker image references. The Harbor project is appended separately as `rsmon`. ## systemd @@ -113,9 +118,17 @@ has only `CAP_NET_RAW` for ICMP checks. | `WORKER_CLUSTER_PEERS` | no | none | Comma-separated `node@host:port` peers. | | `WORKER_CLUSTER_DATA_DIR` | with cluster | none | Persistent Raft state directory. | -The public liveness endpoint is `GET /healthz`. `rsmon-worker health` checks the -configured control plane's `/up` endpoint and is suitable for container health -checks. +The public liveness endpoint is `GET /healthz`; the image probes it with +`rsmon-worker liveness`. `rsmon-worker health` separately checks the configured +control plane's `/up` endpoint for connectivity diagnostics. + +## Implementation Documentation + +Worker architecture, protocol, inventory, private-worker isolation, network +diagnostics, web console, and critical-cluster work are specified in +[`docs/README.md`](docs/README.md). These documents replace worker-owned planning +material formerly kept in the RSMon control-plane repository and include a +source migration ledger. ## Security diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..9907524 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,72 @@ +# Worker Implementation Documents + +This directory is the implementation authority for the standalone RSMon worker. +It converts the worker-related plans formerly kept in the RSMon control-plane +repository into contracts tied to this repository's packages and tests. + +These documents distinguish three states: + +- **Implemented**: wired into `cmd/rsmon-worker` and covered by tests. +- **Partial**: code exists, but a named runtime path or acceptance criterion is + missing. +- **Planned**: no production path exists yet. The document identifies the + package, protocol, dependency, and tests required to implement it. + +Plans do not override code. When a document and the current implementation +disagree, update both in the same change or mark the discrepancy explicitly. + +## Implementation Index + +| Document | Scope | Current state | +| --- | --- | --- | +| [architecture.md](architecture.md) | Process boundaries, ownership, runtime composition | Implemented with partial subsystems | +| [control-plane-protocol.md](control-plane-protocol.md) | WebSocket frames, leases, config, compatibility APIs | Implemented | +| [tasks-and-notifications.md](tasks-and-notifications.md) | Normal checks and delegated notification execution | Implemented with provider gaps | +| [web-console-and-observability.md](web-console-and-observability.md) | Local UI, auth, inventory view, host telemetry | Partial | +| [inventory.md](inventory.md) | Worker host discovery and control-plane inventory projection | Partial | +| [private-workers.md](private-workers.md) | Customer worker isolation, credentials, onboarding | Partial | +| [network-diagnostics.md](network-diagnostics.md) | Confirmation checks and dedicated diagnostic tasks | Partial | +| [critical-check-cluster.md](critical-check-cluster.md) | Raft-backed dispatchless critical checks | Scaffold only | +| [implementation-roadmap.md](implementation-roadmap.md) | Ordered repository work packages and release gates | Active | +| [source-plan-migration.md](source-plan-migration.md) | Source-to-target conversion ledger and resolved conflicts | Complete mapping | + +## Ownership Boundary + +This repository owns: + +- worker process startup, configuration, and graceful shutdown; +- the control-plane client and wire types; +- check and notification execution; +- local web console, local state, inventory collection, and host metrics; +- worker-to-worker transport and Raft state; +- Docker, Compose, systemd, and release-image packaging. + +The RSMon control-plane repository owns: + +- users, accounts, monitors, contacts, billing, and RBAC; +- PostgreSQL task production, selection, leasing, retry, and dead-letter state; +- worker registration and authorization policy; +- accepted-result application and VictoriaMetrics persistence; +- customer-facing fleet, inventory, diagnostics, and incident APIs. + +The worker must not receive control-plane PostgreSQL, Valkey, or +VictoriaMetrics credentials. Normal results, host telemetry, inventory, and +diagnostics are reported over authenticated protocol messages for validation +and persistence by the control plane. + +## Required Quality Gates + +Run before merging worker changes: + +```bash +make check +docker build -t rsmon-worker:test . +docker run --rm rsmon-worker:test --version +docker compose config +``` + +Features that alter protocol messages require compatibility tests in +`internal/wire` and runner protocol tests in `internal/distworker`. Raft work +requires deterministic FSM tests and a multi-node fault test. Security-sensitive +features require negative tests for account scope, arbitrary target rejection, +credential redaction, and unauthenticated access. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..821db98 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,85 @@ +# Worker Architecture + +## Runtime + +`cmd/rsmon-worker/main.go` composes three independently stoppable subsystems: + +1. `internal/distworker`: persistent control-plane connection, normal task + execution, selfcheck, notification delivery, and server metric reports. +2. `internal/webapp`: authenticated local operator console and SQLite-backed + local state. +3. `internal/workercluster`: optional worker-to-worker Raft transport and FSM. + +All three share one cancellation context. The worker can run without the web +console using `--no-web`; the cluster is disabled unless +`WORKER_CLUSTER_ENABLED=true`. + +## Two Execution Paths + +The two check paths are deliberately separate. + +| Path | Work origin | Ownership | Worker implementation | +| --- | --- | --- | --- | +| Normal checks and notifications | Control-plane PostgreSQL `tasks` rows | Exact lease token and expiry held by control plane | `internal/distworker` | +| `distributed_critical` | Signed config adopted by a worker Raft cluster | Raft FSM incident and outbox metadata | `internal/workercluster` plus a planned scheduler bridge | + +Normal work always uses control-plane dispatch and leases. The critical path +must never use a normal task row, job assignment, or lease. Adding the critical +path must not change normal protocol behavior. + +## Package Boundaries + +| Package | Responsibility | Must not do | +| --- | --- | --- | +| `internal/wire` | JSON-compatible protocol types | Network or persistence work | +| `internal/distworker` | Connections, queues, pools, execution orchestration | Direct control-plane database writes | +| `internal/checkexec` | Database-free check dispatch | Scheduling or result persistence | +| `checks/*` | Protocol-specific probes | Worker selection or account policy | +| `internal/webapp` | Local UI, sessions, audit, local snapshots | Become the fleet source of truth | +| `internal/workercluster` | Raft transport, log, FSM, snapshots, membership | Store raw samples or plaintext credentials | +| `internal/sender` | Provider calls shared by worker executors | Own retries or durable task state | + +New host discovery should move into `internal/inventory`; the web console may +read it through an interface but must not remain the collector's owner. New +network probes should live in `internal/netdiag` and use an allowlisted target +provided by a validated control-plane task. + +## Current Status + +| Capability | State | Evidence | +| --- | --- | --- | +| WebSocket normal check execution | Implemented | `internal/distworker/client.go`, `runner.go` | +| Check kinds | Implemented | `internal/checkexec/exec.go`, `checks/*` | +| Delegated notifications | Partial | email, Telegram, webhook, Mattermost implemented; SMS/voice unsupported | +| Host telemetry to control plane | Implemented | `internal/distworker/server_metrics.go` | +| Local web console | Implemented | `internal/webapp/server.go`, `routes.go` | +| Local inventory and status pages | Partial | collectors exist but are not started by `Server.Start` | +| Private-worker scheduler isolation | Implemented in shared selector code | `app/models/worker_node.go`, `task_selector.go`, `check_jobs.go` | +| Private-worker bootstrap and worker-side scope verification | Planned | no one-time bootstrap or signed config path | +| Cross-worker confirmation | Partial | peer selfcheck and shared model logic exist; no rich diagnostic tasks | +| Raft membership and persistence | Partial | `internal/workercluster` | +| Dispatchless critical checks | Planned | no scheduler, signature adoption, quorum incident engine, or outbox executor | + +## Data Rules + +- Credentials delivered in `init`/`config` stay in memory and are never written + to the local SQLite store, Raft log, or snapshots. +- Raw check samples and high-volume telemetry never enter Raft. +- Local UI state is not authoritative control-plane state. +- A normal result is not valid without the current exact lease token. +- Any worker-originated account, server, monitor, or target identifier is + untrusted until the control plane validates it against the authenticated + worker. +- A worker must not execute a synthetic health check whose target is itself. + +## Runtime Completion Criteria + +- SIGTERM drains the runner, stops collectors, closes listeners, shuts down + Raft, and closes SQLite without exceeding the service stop timeout. +- A web-console failure is reported and causes an intentional process policy; + it must not silently leave a partially healthy process. +- Startup rejects malformed `WORKER_URL`, incomplete auth credentials, invalid + cluster settings, and unwritable data directories before accepting work. +- `GET /healthz` reports process-local liveness. Control-plane reachability is + a separate readiness/selfcheck signal and must not make a healthy container + fail its local liveness probe. diff --git a/docs/control-plane-protocol.md b/docs/control-plane-protocol.md new file mode 100644 index 0000000..bfb7efe --- /dev/null +++ b/docs/control-plane-protocol.md @@ -0,0 +1,156 @@ +# Control-Plane Protocol + +## Authority + +`internal/wire/types.go` is the executable schema. New fields must be optional +during rollout, and new branches require decode tests against both old and new +frames. Version strings are informational; explicit capabilities select +protocol features. + +The worker connects to: + +```text +GET /worker?token= +``` + +`/api/worker` and the HTTP jobs/results APIs remain compatibility paths. New +workers use WebSocket task envelopes. + +## Frame Model + +Every frame is a `wire.WorkerMessage` with `kind` and one active content +branch. + +| Direction | `kind` | Active branch | State | +| --- | --- | --- | --- | +| Server to worker | `init` or `config` | `init` | Implemented | +| Server to worker | `task` | `task_envelope` | Implemented | +| Worker to server | `result` | `result` | Implemented | +| Worker to server | `result` | `notification_result` | Implemented | +| Worker to server | `result` | `server_metric` | Implemented | +| Worker to server | `heartbeat` | `heartbeat` | Implemented | +| Either | `error` | `error` | Implemented | +| Worker to server | `result` | inventory report | Planned | +| Server to worker | cluster config/witness | dedicated typed branch | Planned | +| Server to worker | diagnostic task | new task-envelope variant | Planned | + +The legacy top-level `task` and `notification_task` branches are accepted for +rollout compatibility. If a frame includes both a valid current +`task_envelope` and a legacy branch, the worker executes only +`task_envelope`. The current runner ignores malformed or unsupported task +branches and does not yet compare the envelope job ID with the inner job ID; +strict rejection and reporting are P1 work below. + +## Initialization And Refresh + +`wire.WorkerInit` supplies runtime values owned by the control plane: + +- worker ID, region, advertised URL, capabilities, and concurrency; +- allowed notification methods and account IDs; +- optional linked server ID for host metrics; +- LLM endpoints; +- scoped notification credentials and system contacts; +- peer workers for selfcheck/cluster-adjacent behavior. + +The worker clamps supplied concurrency to its local maximum. Credentials are +replaced atomically in memory on refresh. Removed credentials must become +unavailable immediately after the refresh is applied. + +Private-worker hardening will add an immutable worker account ID, config +version, expiry, and signature. Until then the executable trusts the +authenticated control plane to send a correctly scoped config; server-side +selection remains the primary isolation boundary. + +## Normal Task Envelope + +`wire.TaskEnvelope` is a tagged union: + +```json +{ + "kind": "task", + "task_envelope": { + "type": "check", + "job_id": "uuid", + "check": { + "job_id": "uuid", + "lease_token": "per-lease-secret", + "check_id": 123, + "monitor_id": 456, + "kind": "http", + "host": "example.com", + "url": "https://example.com", + "interval": 60, + "settings": {} + } + } +} +``` + +`type=notification` activates `notification` instead. Job IDs in the envelope +and active branch must match once P1 validation lands. Today a task without a +recognized populated branch is ignored without execution. + +## Result Invariants + +- Echo `job_id` and the exact `lease_token` from the task. +- Send one terminal result per execution attempt. +- Never retry a result by executing the task again. Result transport retries + resend the same terminal report. +- Treat duplicate terminal acknowledgements as success. +- Do not infer task acceptance from a WebSocket write alone; durable ownership + remains on the control plane until it validates the result. +- Bound error strings and provider responses before transmission. + +## Host Metrics + +`wire.ServerMetricReport` is sent only after `WorkerInit.ServerID` is present. +The worker collects locally and sends bounded snapshots. The control plane +validates worker/server/account ownership and persists both the latest cache +and VictoriaMetrics points. The worker does not have TSDB credentials. + +## Protocol Work Packages + +### P1: Conformance Tests + +Files: + +- `internal/wire/types_test.go` +- `internal/distworker/runner_protocol_test.go` + +Tests: + +- decode every current frame branch; +- prefer the current envelope over duplicate legacy branches; +- reject mismatched envelope and inner job IDs; +- preserve unknown optional fields during compatible rollout; +- reject missing lease tokens before execution result submission; +- atomically replace config and credentials. + +### P2: Safe Token Rotation + +Current `Runner.RotateToken` closes the runner stop channel. Replace this with a +connection-scoped cancellation path that stores the new token, closes only the +active WebSocket, and reconnects without terminating worker services. + +Acceptance: rotating from the web console produces a reconnect using the new +token while the HTTP listener, collectors, and optional cluster stay running. + +### P3: Signed Private-Worker Config + +Add to the init/config branch: + +- `account_id` for private workers; +- monotonic `config_version`; +- `issued_at` and `expires_at`; +- signature key ID and Ed25519 signature over canonical payload bytes. + +Reject regressions, invalid signatures, expired config, and account changes. +Keep the last valid config only until its expiry; do not silently accept an +invalid replacement. + +### P4: New Typed Branches + +Inventory, diagnostic, and critical-cluster messages each receive a dedicated +wire type. Do not tunnel them through `event` or arbitrary `json.RawMessage`. +Each branch must define payload limits, account/target validation ownership, +idempotency, and compatibility behavior before implementation. diff --git a/docs/critical-check-cluster.md b/docs/critical-check-cluster.md new file mode 100644 index 0000000..dcd8ae5 --- /dev/null +++ b/docs/critical-check-cluster.md @@ -0,0 +1,188 @@ +# Dispatchless Critical-Check Cluster + +## Status + +`internal/workercluster` is a scaffold, not a production critical-check engine. +It currently provides Hashicorp Raft, bbolt log/stable storage, snapshots, +HTTP-based authenticated transport, bootstrap/join/membership operations, log +entry types, a small FSM, and operator status. + +It does not yet execute `distributed_critical` checks, verify signed config, +evaluate observation/region/notification quorum, encrypt snapshots, deliver a +commit-backed outbox, or consume an external witness report. + +The hardcoded test-config endpoint and startup flag are development-only and +must stay disabled in production. + +## Non-Negotiable Separation + +`distributed_critical` is a distinct path: + +- no control-plane job dispatch; +- no PostgreSQL task row; +- no task lease or single worker owner; +- every eligible observer executes the adopted check deterministically; +- raw samples go to the control plane/TSDB outside Raft; +- classified observations, compact incidents, and outbox metadata enter Raft; +- normal checks and normal notifications continue unchanged. + +The control plane signs configuration, receives replay, and acts as an external +witness. It is never a Raft voter. + +## Topology + +- Production clusters have 3 or 5 voters, never an even count. +- Voters remain within a bounded-latency topology. Remote regions use observer + nodes unless measured RTT supports the configured election timeout. +- Nodes may be `voter`, `observer`, or `voter+observer`. +- Observer count affects observation policy, not Raft election quorum. +- A node without durable, real-fsync storage cannot be a voter. + +Single-voter bootstrap is temporary. The cluster must not execute customer +critical checks until at least three voters are healthy and the signed +observer set is committed. + +## Raft State + +The FSM contains only: + +- monotonic config version and adopted critical-check definitions; +- versioned observer set; +- compact incident state and committed observations; +- bounded notification outbox metadata and idempotency tombstones; +- compact member diagnostics and partition/witness state; +- tenant isolation policy and applied-version index. + +It never contains raw probe samples, response bodies, provider bodies, +plaintext credentials, credential envelopes, or normal task leases. + +`raft-boltdb/v2` is only the local Raft `LogStore` and `StableStore`; it is not +the application model. Snapshots are produced by the FSM and must be versioned, +checksummed, and encrypted at rest before production use. + +## Log Entries + +Allowed application entry kinds: + +- `config.adopt` +- `observer_set.update` +- `incident.observe` +- `incident.transition` +- `outbox.enqueue` +- `outbox.delivered` +- `outbox.ack` +- `partition.report` +- `membership.propose_add` +- `membership.demote` +- `membership.remove` +- `diagnostics.update` + +No `task.lease.*`, retry, completion, or dead-letter entries belong in this +package. + +## Signed Config Adoption + +The control plane sends an Ed25519-signed canonical payload containing: + +- cluster and tenant IDs; +- monotonic config version and expiry; +- critical checks and incident policies; +- proposed observer set, role/region map, and content hash; +- signing key ID and credential-envelope references. + +Every node verifies signature, cluster identity, expiry, monotonic version, +observer-set hash, and supported schema. The leader proposes `config.adopt` and +`observer_set.update`. No observer executes the new config until both commits +are applied locally. Invalid config is rejected and reported through compact +diagnostics; there is no silent downgrade. + +## Scheduling And Observation + +For each adopted check, eligible observers derive the same interval boundary +from `epoch + n*interval`. Per-worker jitter is deterministic from worker ID, +check ID, observer-set version, and interval number. At each tick: + +1. Run the underlying probe through `internal/checkexec` outside Raft. +2. Send raw metrics through the normal control-plane metrics path. +3. Classify to `ok`, `warn`, `down`, or `unknown` using signed policy. +4. Propose `incident.observe` to the leader with committed timestamp and + observer/config versions. + +Followers forward proposals or return a typed not-leader response containing +the current leader. They never silently drop observations. + +## Four Quorums + +- `raft_quorum`: majority of voters required to commit. +- `observation_quorum`: observers agreeing within the observation window. +- `region_quorum`: represented regions agreeing on state. +- `notification_quorum`: region agreement required for first customer alert. + +All names remain distinct in code, metrics, configuration, and logs. + +## Deterministic Incident FSM + +`FSM.Apply` must perform no I/O, network calls, wall-clock reads, randomness, +or provider calls. It uses only committed payload values and prior state. + +Incident lifecycle is `clear -> observing -> open -> resolving -> clear`. +Policy includes confirmation count, observation window, classification, +minimum incident dwell, transition-rate suppression, cooldown, and outbox retry +metadata. Apply enforces bounded state and deterministic idempotency keys. + +The current FSM merely records the latest observation and accepts externally +constructed transitions. Replace this placeholder with deterministic policy +evaluation and tests before running real checks. + +## Commit-Before-Notify + +Strict order: + +1. Probe and classify outside Raft. +2. Commit observation at `raft_quorum`. +3. FSM deterministically commits incident transition and outbox metadata. +4. After the committed outbox entry is visible, an executor calls the provider + outside Raft. +5. Provider acknowledgement produces `outbox.delivered`; retry/failure produces + bounded metadata updates. + +Provider credentials stay in node-local memory/secure storage and are never +part of an entry or snapshot. Delivery uses stable channel idempotency keys to +survive leader failover. + +## Partitions And Witness + +Partition states are `steady`, `degraded`, `partitioned`, `healing`, +`split_brain_detected`, and `witness_only`. Loss of `raft_quorum` freezes +incident transitions and outbox creation. A cluster must never fabricate an +open or recovery while partitioned. + +The external witness reports observed leader/term, reachable voters and +observers, split-brain flag/time, and report time. It can alert and inform state +but cannot commit or fabricate incidents. + +## Ordered Implementation + +1. Remove production exposure of debug config application. +2. Complete versioned FSM types, command validation, and deterministic tests. +3. Add mTLS identity and safe one-claim cluster bootstrap. +4. Add signed config and observer-set adoption. +5. Add deterministic scheduler and checkexec bridge in shadow mode. +6. Implement observation aggregation, incident policy, and idempotency. +7. Implement encrypted snapshots and restore/migration tests. +8. Add metadata outbox executor and failover-safe delivery. +9. Add external witness, replay, metrics, and operational runbooks. +10. Run synthetic 3/5-node fault campaigns before any customer check. + +## Release Gates + +- Two independent bootstraps for one cluster are detected and rejected. +- Changes that leave fewer than three or an even number of voters are rejected. +- Leader failover completes within twice the configured election timeout. +- A minority partition commits zero incident transitions. +- Loss of observation quorum with intact Raft quorum sends no notification. +- Snapshot inspection finds no raw samples or credential material. +- Signature mutation and observer-set version mismatch prevent execution. +- Killing the leader between provider acknowledgement and metadata commit does + not create duplicate customer-visible notification effects. +- Normal WebSocket tasks and notifications continue throughout cluster tests. diff --git a/docs/implementation-roadmap.md b/docs/implementation-roadmap.md new file mode 100644 index 0000000..f9995b9 --- /dev/null +++ b/docs/implementation-roadmap.md @@ -0,0 +1,165 @@ +# Worker Implementation Roadmap + +This roadmap orders work by dependency and risk. A package existing does not +make a work package complete; its release gate must pass in a wired process. + +## R0: Distribution Reliability + +State: active. + +Worker repository: + +- publish valid Harbor references after normalizing a configured registry URL; +- use local `/healthz` for container liveness; +- verify amd64 and arm64 image startup, Chromium availability, and version + metadata; +- add a package/install smoke test for Docker and systemd artifacts; +- document immutable SHA and release tags as production defaults. + +Gate: a push publishes `sha-<12>` and `latest` manifests for both platforms, +and a container remains healthy when the control plane is unavailable. + +## R1: Runtime Correctness + +State: partial. + +Worker repository: + +- wire local inventory and metrics collector lifecycle into web server start + and shutdown; +- validate full HTTP config, including `WORKER_URL`, in main startup; +- make token rotation reconnect rather than stop the runner; +- define process policy when the web listener exits unexpectedly; +- add WebSocket reconnect, drain, duplicate frame, and backpressure tests. + +Gate: collectors populate real pages, rotation preserves all subsystems, and +SIGTERM leaves no listener, collector, task, or SQLite goroutine behind. + +## R2: Protocol And Credential Hardening + +State: partial. + +Worker repository: + +- complete frame conformance tests; +- add signed config version/account identity; +- enforce task account and credential scope locally; +- add bounded delegated-delivery audit to the local console; +- support per-account webhook/Mattermost credentials after control-plane wire + support exists. + +Control-plane dependency: + +- signed config producer and key rotation; +- persisted account-scoped webhook/Mattermost credentials; +- lease/task account fields treated as immutable during result application. + +Gate: cross-account fixtures fail before network execution and no credential +appears in worker logs, SQLite, snapshots, or protocol error payloads. + +## R3: Private-Worker Lifecycle + +State: planned beyond scheduler isolation. + +Worker repository: + +- one-time bootstrap exchange and atomic secret storage; +- token handoff and revocation handling; +- stale/expired signed-config behavior; +- clear disabled/revoked UI state; +- optional mTLS client identity. + +Control-plane dependency: + +- customer worker CRUD and billing entitlement; +- one-time bootstrap token state; +- immutable account binding and signed config; +- audit and revocation acknowledgement. + +Gate: a customer can install, connect, execute only their work, rotate, disable, +revoke, and uninstall without an operator admin secret. + +## R4: Host Inventory And Console + +State: partial. + +Worker repository: + +- extract `internal/inventory` and start process/host collection; +- add typed bounded inventory report; +- add Docker/Compose, nginx, systemd, and address collectors in stages; +- add read-only Compose status/logs before mutation; +- correct local interval metrics and add optional hardware sections. + +Control-plane dependency: + +- worker inventory report ingestion and source-aware reconciliation; +- server/worker/account ownership validation; +- deploymentd comparison and monitor suggestion UI. + +Gate: partial failures cannot delete inventory, worker and deploymentd sources +coexist, and reports contain no environment or credential values. + +## R5: Dedicated Network Diagnostics + +State: planned; normal confirmation is already partial. + +Worker repository: + +- add diagnostic wire branch and `internal/netdiag`; +- ship HTTP GET/HEAD and unauthenticated SSH handshake first; +- add TCP/DNS, then separately gate traceroute/MTR; +- implement SSRF, redirect, timeout, output, and rate protections. + +Control-plane dependency: + +- monitor-bound target normalization and signed target hash; +- diagnostic task production, storage, RBAC, rate limits, and UI; +- private-worker source eligibility. + +Gate: no task can probe outside its authorized monitor target and all results +are bounded, structured, and credential-free. + +## R6: Public Check Economy + +State: planned. + +Depends on R2, R3, and diagnostic-grade target protection. Public work uses a +separate signed grant and safe HTTP methods, never widened account scope. + +Gate: private targets and credential-bearing requests are impossible under DNS +rebinding and redirects; accounting remains idempotent under result replay. + +## R7: Critical-Check Cluster + +State: Raft scaffold only. + +Worker repository: + +- secure bootstrap and mTLS identity; +- signed config adoption and observer-set versioning; +- deterministic scheduler and check executor bridge; +- deterministic incident/quorum FSM; +- encrypted snapshots and restore; +- metadata outbox, witness, replay, and metrics; +- 3/5-node fault campaigns. + +Control-plane dependency: + +- critical-check config/signing service; +- credential-envelope service; +- witness and replay endpoints; +- customer incident projection and audit. + +Gate: all release gates in +[critical-check-cluster.md](critical-check-cluster.md) pass while normal tasks +continue without behavioral change. + +## Documentation Rule + +Every completed work package updates: + +- the relevant implementation document's current-state section; +- this roadmap state and gate evidence; +- root configuration examples when environment or deployment changes; +- the source migration ledger if a control-plane plan is superseded. diff --git a/docs/inventory.md b/docs/inventory.md new file mode 100644 index 0000000..485f801 --- /dev/null +++ b/docs/inventory.md @@ -0,0 +1,165 @@ +# Worker Host Inventory + +## Purpose And Boundary + +Inventory connects checks to the host, service, Compose project, domain, and +deployment that they observe. The worker collects facts; the RSMon control +plane validates identity, applies account scope, reconciles lifecycle, and +persists the inventory projection. + +The control-plane inventory entities remain `Server`, `ServerIp`, `Site`, +`Deployment`, `Domain`, `Repo`, and `site_repos`. Their API and rstuff stream +projection are control-plane concerns. This repository owns only local +discovery and worker-originated reports. + +## Existing Collectors + +Current state: + +- `internal/webapp/inventory.go` discovers Linux processes, command lines, + working directories, TCP listeners, and basic resource facts. +- snapshots are stored in the local web-console SQLite database; +- `internal/distworker/server_metrics.go` reports bounded process/network host + snapshots, but not normalized application inventory; +- external `deploymentd` remains the implemented authoritative collector for + nginx, Docker Compose, and host inventory on the control plane. + +The worker does not currently send an inventory report to the control plane. +It must not claim deploymentd parity until the protocol, ingestion, and +reconciliation tests below are complete. + +## Target Package Layout + +Move collection ownership out of `internal/webapp`: + +```text +internal/inventory/ + collector.go orchestration and partial-success envelope + process_linux.go procfs process and listener discovery + docker.go Docker and Compose discovery behind capability + nginx.go read-only nginx virtual-host discovery + systemd.go allowlisted unit discovery + normalize.go stable IDs and control-plane report conversion + store.go optional local snapshot interface +``` + +The web console consumes a read-only snapshot interface. The runner consumes a +bounded report interface. Collectors return per-section errors; one failed +section does not discard successful sections. + +## Report Contract + +Add a typed `WorkerInventoryReport` branch to `internal/wire`: + +```json +{ + "kind": "result", + "inventory": { + "schema": "rsmon.worker.inventory.v1", + "report_id": "uuid", + "server_id": 42, + "collected_at": "2026-07-13T15:00:00Z", + "full_snapshot": true, + "host": {}, + "addresses": [], + "processes": [], + "compose_projects": [], + "nginx_sites": [], + "systemd_units": [], + "errors": [{"section": "docker", "code": "unavailable"}] + } +} +``` + +Constraints: + +- `report_id` is stable across transport retries and unique across collection + runs. +- `server_id` comes from authenticated `WorkerInit.ServerID`, never local env. +- A full snapshot permits missing-item reconciliation; partial reports do not. +- Section and total item counts, strings, labels, and serialized bytes are + bounded before enqueue. +- Environment variable values, file contents, process environments, container + secrets, and Compose `.env` values are never included. +- The control plane rejects a report when the worker is not linked to the + server or the private worker's account does not own it. + +## Normalization + +Stable local identity keys: + +- process app: executable plus canonical working directory; +- Compose site: Docker Compose project name plus canonical project directory; +- Compose deployment: project plus service name; +- nginx deployment: canonical config path; +- domain: normalized lowercase hostname without trailing dot; +- systemd service: unit name. + +Container IDs and PIDs are observations, not stable entity IDs. Never use them +as the sole upsert key. Every deployment includes `last_seen_at`; only a full +successful section can mark previously observed entities missing. + +## Collector Requirements + +### Processes + +- Group related processes deterministically by executable, working directory, + and parent relationship. +- Parse TCP and UDP listeners for IPv4 and IPv6. +- Do not read another process's environment. +- Exclude kernel threads and the worker itself from application suggestions. + +### Docker And Compose + +- Disabled without an explicit Docker capability. +- Prefer Docker/Compose JSON output and labels; do not parse human tables. +- Collect project, service, image, state, ports, mount paths, health, and + Traefik host rules. Exclude environment values and registry credentials. +- Treat Docker socket access as root-equivalent and report the capability to + the operator. + +### Nginx + +- Read only configured allowlisted roots. +- Collect config path, listen values, server names, auth presence, root path, + and proxy presence. Do not send certificate private-key paths or contents. + +### Host And Server IPs + +- Collect hostname, OS/kernel summary, and valid non-loopback addresses. +- Do not guess a primary address; report interface and route metadata so the + control plane can apply policy. + +## Relationship To deploymentd And rstuff + +Worker reports and deploymentd ingest are complementary producers of the same +control-plane projection. During migration: + +- deploymentd remains authoritative for sections it currently reports; +- worker inventory is feature-gated per server; +- every row records source and source report ID; +- reconciliation occurs per source and section to prevent one producer from + deleting another producer's observations; +- the rstuff `rsmon.inventory.v1` stream remains a control-plane output/input, + not a worker protocol. + +## Delivery Phases + +1. Start and stop the existing process collector; move it behind an interface. +2. Add typed report and control-plane ingestion for process/host/address data. +3. Add read-only Docker and Compose discovery. +4. Add nginx and systemd discovery. +5. Add source-aware reconciliation and deploymentd comparison mode. +6. Enable monitor suggestions only after operators confirm discovered targets. + +## Acceptance Tests + +- A worker restart preserves no false stable identity based on PID/container ID. +- A failed Docker section does not erase successful nginx or process data. +- Partial reports never mark absent deployments missing. +- Full reports reconcile only rows owned by the same source and section. +- Cross-account and worker/server mismatch reports are rejected. +- Fixture snapshots prove no environment values or credential-like fields are + serialized. +- Worker and deploymentd fixtures normalize equivalent Compose/nginx entities + to the same control-plane identity without deleting each other's rows. diff --git a/docs/network-diagnostics.md b/docs/network-diagnostics.md new file mode 100644 index 0000000..7b10d33 --- /dev/null +++ b/docs/network-diagnostics.md @@ -0,0 +1,127 @@ +# Network Diagnostics + +## Current Capability + +Two different features must not be conflated: + +1. **Cross-worker confirmation** is partially implemented. The control plane + can assign the same failed check to another operated worker, and peer + selfcheck/quorum helpers exist in `internal/distworker/peer.go`, + `consensus.go`, and `selfcheck.go`. +2. **Rich diagnostic tasks** (`diag_http`, `diag_ssh`, DNS, TCP, traceroute, + MTR) are not implemented as worker protocol kinds. + +Confirmation remains a normal centrally leased check and does not require a +new executor. Rich diagnostics require typed tasks and structured timing/path +results. + +## Security Boundary + +Diagnostics can become a network scanner. Every diagnostic task must be bound +to a monitor target already authorized and normalized by the control plane. +The worker receives the normalized target plus a signed target hash and refuses +runtime overrides. + +Required restrictions: + +- private worker account must equal task account; +- source worker must not equal a target worker represented by the monitor; +- no arbitrary host, URL, resolver, port range, or shell arguments; +- resolve every hostname locally and reject prohibited IPs before connecting; +- revalidate every HTTP redirect target; +- enforce per-task timeout, output limit, concurrency, and destination rate; +- never return authorization headers, request bodies, response bodies, SSH + credentials, or environment data; +- audit allow and deny decisions on the control plane. + +## Target Package + +```text +internal/netdiag/ + types.go + validate.go + http.go + ssh.go + tcp.go + dns.go + traceroute_linux.go + mtr.go +``` + +`internal/netdiag` accepts a validated immutable request and returns a bounded +result. It does not select workers, query monitors, persist data, or send +protocol frames. `internal/distworker` owns task dispatch and result transport. + +## Protocol + +Extend `wire.TaskEnvelope` with `type=diagnostic` and a dedicated diagnostic +branch. Do not encode diagnostics as check settings or untyped events. + +Common request fields: + +- job ID and exact lease token; +- account, monitor, source worker, optional target worker, and region IDs; +- kind and scheduling reason (`periodic`, `on_demand`, `failure_triggered`); +- normalized target and target hash; +- timeout and kind-specific bounded options. + +Common result fields: + +- job ID, lease token, kind, source and target IDs; +- start/finish time and total latency; +- `ok` or `err` status; +- normalized error class; +- bounded kind-specific data. + +Error classes are `timeout`, `dns_error`, `tcp_refused`, `tcp_reset`, +`tls_handshake`, `tls_cert`, `http_status`, `auth_required`, `auth_failed`, +`protocol_error`, `target_denied`, `unsupported_kind`, and `internal_error`. + +## Delivery Phases + +### D1: HTTP Diagnostics + +Implement GET/HEAD only. Capture DNS, connect, TLS, first-byte, total timing, +status, final URL, bounded redirect chain, selected response headers, response +size, TLS version/cipher, and certificate fingerprints. Do not return body +content. Clamp total timeout to 30 seconds. + +### D2: SSH Diagnostics + +Perform banner, key exchange, and host-key inspection without authentication. +Return DNS/connect/KEX timing, banner, selected algorithms, and SHA-256 host-key +fingerprint. Password and private-key authentication are not part of v1. + +### D3: TCP And DNS + +TCP performs one destination connect and distinguishes timeout, refused, and +reset. DNS resolves only the monitor hostname with the system or explicitly +allowlisted resolver and returns bounded A/AAAA/CNAME answers and timing. + +### D4: Traceroute And MTR + +Linux-only, capability-gated, and disabled for private workers until abuse +review. Use a native bounded implementation or a fixed executable with fixed +arguments; never pass user strings to a shell. Return a maximum hop count and +bounded probes per hop. + +## Relationship To Raft + +Normal diagnostic tasks remain centrally leased. Do not replicate raw +diagnostic results in Raft. If duplicate failure-triggered diagnostics become a +measured problem, the critical cluster may replicate only compact dedup +metadata; this is not a prerequisite for diagnostic v1. + +## Acceptance Tests + +- Existing confirmation excludes the original failing worker and handles + duplicate results idempotently. +- Diagnostic tasks cannot target a host or port different from their signed + monitor target. +- HTTP redirects to prohibited addresses are rejected. +- Private workers reject cross-account tasks before opening a socket. +- HTTP and SSH fixture servers produce deterministic phase timings and bounded + metadata. +- Timeouts cancel DNS/connect/TLS/read work and leave no goroutine behind. +- Unsupported kinds return a structured terminal result. +- Result serialization contains no credentials or response body. diff --git a/docs/private-workers.md b/docs/private-workers.md new file mode 100644 index 0000000..c63c704 --- /dev/null +++ b/docs/private-workers.md @@ -0,0 +1,147 @@ +# Private Workers + +## Model + +Operated and customer-operated workers run the same binary and protocol. The +control plane assigns trust and scope. A private worker is untrusted for every +account except its own and cannot widen its own scope through capabilities or +protocol fields. + +Current shared scheduler code already supports restrictive account selection: + +- `app/models/worker_node.go` derives allowed check and notification accounts; +- `app/models/check_jobs.go` filters check leasing; +- `app/models/task_selector.go` filters generic task leasing; +- a nonempty account list is restrictive; an empty list means platform worker + and is never available to a customer worker. + +This is only one isolation layer. The executable still trusts the authenticated +control plane's init/task payload and private-worker onboarding is incomplete. + +## Required Invariants + +- A private worker has one immutable `account_id` in signed runtime config. +- Every normal task includes `account_id`; the worker rejects mismatch before + execution and reports a protocol error without touching the target. +- Credentials are filtered by account and method on the control plane, then + checked again by exact credential ID in the worker. +- Private workers do not receive another account's monitor, contact, LLM, peer, + inventory, or notification data. +- A private worker cannot become operated by changing local config or reported + capabilities. +- Results cannot choose their account. The control plane applies them against + the leased task and authenticated worker. +- Local UI data stays on the worker unless a typed, bounded protocol explicitly + permits upload. + +## Onboarding + +The current production path uses a worker token created by an administrator. +The customer self-service target is: + +1. Control plane creates a disabled private-worker record and a one-time, + 15-minute bootstrap token. +2. Installer writes control-plane URL and bootstrap token to a mode-0600 + temporary environment file. +3. Worker exchanges it over TLS for a long-lived worker token and signed + immutable account identity. +4. Worker writes the long-lived token to the configured secret file with mode + 0600, removes the bootstrap token, and reconnects. +5. Control plane activates the worker only after the first authenticated + heartbeat and valid capability report. + +The worker must never receive the control-plane registration admin secret. +Bootstrap replay, expiry, worker-ID mismatch, or account mismatch fails closed. + +## Token Rotation + +Rotation is a two-token handoff: + +1. Control plane issues a replacement token and keeps the old token valid for a + short bounded overlap. +2. Worker atomically writes the replacement secret. +3. Worker closes only the active control-plane connection and reconnects. +4. Successful authentication acknowledges rotation; control plane revokes the + old token. +5. Failure retains the old token until overlap expires and emits an operator + warning. + +The current `Runner.RotateToken` terminates the runner by closing its stop +channel. This must be fixed before claiming unattended rotation. + +## Runtime Config Authentication + +Private-worker init/config payloads add: + +- worker ID and immutable account ID; +- monotonic config version; +- issue and expiry timestamps; +- allowed check kinds, notification methods, and public-task policy; +- credential-set hash; +- Ed25519 signature and key ID. + +The worker pins the control-plane verification key at bootstrap. It rejects +signature failure, downgrade, expiry, account change, and unknown critical +fields. Credentials remain in memory and are cleared when their signed scope +expires. + +## Public Checks + +Cross-account public-check execution is not the same as normal private scope. +It requires a separately signed `public_tasks` grant with: + +- explicit opt-in by worker owner; +- allowed methods limited to safe public probes, initially HTTP GET/HEAD; +- no credentials, custom authorization headers, request bodies, private IP + targets, or notification tasks; +- independent concurrency and rate limits; +- SSRF validation after every DNS resolution and redirect; +- auditable reward/usage identity owned by the control plane. + +Do not represent public permission by making the private worker's account list +empty or adding arbitrary account IDs. + +## Deployment Modes + +The systemd and Compose packages are both supported. Capabilities, not install +type, control host access: + +- base: normal network checks and delegated notifications; +- host metrics: procfs and statfs access; +- inventory: read-only process/system discovery; +- Docker discovery: explicit socket access, treated as root-equivalent; +- Compose mutation: separate high-risk capability, disabled by default; +- Raft voter: durable fsync-capable cluster data directory and mTLS transport. + +`CAP_NET_RAW` is granted only for ping/traceroute features. The worker remains +unprivileged otherwise. + +## Worker Self-Monitoring + +A worker must not run a monitor that represents its own process or host. The +control plane excludes it during selection; the worker also rejects a task that +names its own worker/host identity once those fields are signed into the task. +Peer health checks and control-plane selfcheck are separate from customer +monitor execution. + +## Implementation Work Packages + +1. Add signed account/config identity to `internal/wire` and runner state. +2. Validate task account and credential scope locally before dispatch. +3. Implement one-time bootstrap and atomic token storage/rotation. +4. Add worker disable/revoke behavior and visible stale-config state. +5. Add mTLS as an optional first transport, then require it for Raft clusters. +6. Add public-task grant and SSRF-safe executor only after private isolation is + proven. + +## Acceptance Tests + +- A private worker never leases or executes another account's normal task. +- A forged capability or account field cannot widen scope. +- Invalid, expired, downgraded, or differently scoped signed config is rejected. +- Bootstrap tokens are single-use and absent from disk after exchange. +- Rotation reconnects without stopping web, inventory, metrics, or cluster. +- Credential snapshots contain only the worker account and allowed methods. +- Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix + sockets, or a private redirect target. +- Revocation prevents reconnect and clears in-memory credentials. diff --git a/docs/source-plan-migration.md b/docs/source-plan-migration.md new file mode 100644 index 0000000..718d960 --- /dev/null +++ b/docs/source-plan-migration.md @@ -0,0 +1,125 @@ +# Source Plan Migration Ledger + +## Purpose + +This ledger records how worker-related documents from the RSMon control-plane +repository were converted. The source plans are historical input, not runtime +authority. Their worker-owned requirements now live in this repository's +implementation documents; control-plane-owned requirements stay in RSMon. + +Conversion date: 2026-07-13. + +## Direct Worker Documents + +| Former RSMon document | Worker implementation document | Resolution | +| --- | --- | --- | +| `docs/new-worker.md` | root `README.md`, `architecture.md`, `private-workers.md` | Replaced monorepo commands with standalone binary/container/systemd operation | +| `docs/worker-container.md` | root `README.md`, `implementation-roadmap.md` | Distribution ownership moved here; production image tags retained | +| `docs/worker-http-settings.md` | `web-console-and-observability.md`, root `README.md` | Current port/auth behavior retained; Compose-only `WORKER_BIND_IP` clarified | +| `docs/worker-protocol.md` | `control-plane-protocol.md` | Current tagged task envelope is normative; stale jobs/results envelopes rejected | +| `docs/worker-cluster-quickstart.md` | `critical-check-cluster.md` | Existing cluster scaffold separated from unimplemented critical engine | +| `docs/rsmon-distributed-api-plan.md` | `architecture.md`, `control-plane-protocol.md`, `implementation-roadmap.md` | PostgreSQL normal path retained; Redis/Rabbit/NATS queue proposal superseded | +| `docs/plans/worker-notifier-mvp.md` | `tasks-and-notifications.md` | Final `task_envelope` and credential implementation supersede older sibling-field drafts | +| `docs/sessions/monitor-transfer-and-admin-workers.md` | `architecture.md`, `private-workers.md` | Historical operator/session decisions reduced to current ownership and account-isolation rules | + +## Distributed Plans + +| Former RSMon document | Worker implementation document | Resolution | +| --- | --- | --- | +| `docs/distributed/notifications-from-worker.md` | `tasks-and-notifications.md`, `private-workers.md` | Implemented methods/statuses recorded; old credential assumptions replaced | +| `docs/distributed/private-workers.md` | `private-workers.md` | Scheduler isolation marked current; bootstrap, signed config, and mTLS remain explicit work | +| `docs/distributed/network-diagnostics.md` | `network-diagnostics.md` | Current confirmation separated from future rich diagnostic tasks | +| `docs/distributed/worker-to-worker-raft.md` | `critical-check-cluster.md` | Converted to package-specific ordered implementation and release gates | +| `docs/distributed/critical-check-amendments.md` | `critical-check-cluster.md` | No-leases, four-quorum, external-witness, and commit-before-notify rules made normative | +| `docs/distributed/worker-web-app.md` | `web-console-and-observability.md`, `inventory.md` | Existing console recognized; future OAuth/Compose/hardware scope split into staged work | + +## Inventory And Host Documents + +| Former RSMon document | Worker implementation document | Resolution | +| --- | --- | --- | +| `docs/inventory-management.md` | `inventory.md` | Control-plane projection retained as dependency; worker owns collection only | +| `docs/plans/inventory-management.md` | `inventory.md` | deploymentd and worker reports defined as source-aware complementary producers | +| `docs/parity/rstuff-inventory.md` | `inventory.md` | rstuff enums/stream remain control-plane contracts, not worker protocol | +| `docs/parity/deploymentd.md` | `inventory.md` | deploymentd parity becomes fixture normalization and comparison gate | +| `docs/deploymentd-replay-rollout.md` | `inventory.md` | HTTP nonce rollout remains deploymentd-specific; worker uses authenticated typed reports | +| `docs/integrations/rstuff-inventory-stream.md` | `inventory.md` | Valkey stream remains between control plane and rstuff, never worker transport | +| `docs/plans/servers-and-hardware-metrics.md` | `web-console-and-observability.md`, `inventory.md` | Implemented server metric report retained; local hardware extensions staged | +| `docs/server-observability.md` | `web-console-and-observability.md`, `control-plane-protocol.md` | Current wire fields, ownership validation, and TSDB boundary retained | +| `docs/parity/capture.md` | `web-console-and-observability.md`, `implementation-roadmap.md` | Worker-host coverage gaps converted into staged collector work | + +## Diagnostics, Policy, And Product Documents + +| Former RSMon document | Worker implementation document | Resolution | +| --- | --- | --- | +| `docs/network-diagnostics.md` | `network-diagnostics.md` | Implemented confirmation/quarantine baseline retained | +| `docs/plans/network-diagnostics-partial.md` | `network-diagnostics.md` | Partial implementation folded into explicit rich-task phases | +| `docs/notification-task-delivery.md` | `tasks-and-notifications.md`, `control-plane-protocol.md` | Durable queue remains control-plane-owned; attempt execution remains worker-owned | +| `docs/notification-credentials.md` | `tasks-and-notifications.md`, `private-workers.md` | SMTP/Telegram current scope retained; webhook/Mattermost gaps explicit | +| `docs/billing-and-entitlements.md` | `private-workers.md`, `implementation-roadmap.md` | Entitlement remains a control-plane gate, never trusted from worker config | +| `docs/plans/public-checks.md` | `private-workers.md`, `implementation-roadmap.md` | Public tasks use a separate safe grant, not widened account access | +| `docs/plans/plans-and-billing.md` | `private-workers.md` | Plan eligibility remains control-plane policy | +| `docs/check-kinds.md` | `tasks-and-notifications.md` | Worker-supported executor list tied to `internal/checkexec` | +| `docs/plans/uptimerobot-parity.md` | `implementation-roadmap.md` | Worker-owned parity dependencies ordered by security prerequisites | +| `docs/plans/krasichka.md` | none | External Krasichka runtime is not a feature of this worker binary; only protocol compatibility applies | + +## Operations And CI Documents + +| Former RSMon document | New authority | Resolution | +| --- | --- | --- | +| `docs/local-processes.md` | root `README.md` | Standalone development commands replace monorepo process instructions | +| `docs/ci.md` | `.github/workflows/*.yml`, `implementation-roadmap.md` | Worker image build is owned by Gitea Actions here | +| `docs/CI_QUICKSTART.md` | root `README.md` | Standalone make, Docker, and Compose commands are authoritative | + +## Resolved Conflicts + +### Protocol + +Old plans used `type: jobs/results`, separate notification siblings, and +version-string feature detection. Current code uses `kind`, explicit branches, +`task_envelope`, exact lease tokens, and explicit capabilities. The current +model is authoritative. + +### Queue Ownership + +Normal task ownership is PostgreSQL with `FOR UPDATE SKIP LOCKED`. Redis +Streams, RabbitMQ, and NATS are not worker dependencies. The critical path is +dispatchless Raft and cannot reuse normal leases. + +### Credentials + +SMTP and Telegram use scoped pushed credential lists. Webhook and Mattermost +are not yet equivalently persisted per account. SMS and voice are unsupported. +No document may claim universal per-account provider isolation before those +gaps close. + +### Web Console + +The console exists on port 27401 with Basic auth or local bcrypt. The former +draft's port 7401, OAuth-first behavior, and entirely unimplemented status are +obsolete. Collector lifecycle remains a real implementation gap. + +### Inventory + +deploymentd HTTP ingest is currently implemented on the control plane. Worker +inventory is local/partial and has no upstream report yet. Future worker and +deploymentd reports are complementary source-scoped inputs; neither may erase +the other's observations. + +### Private Workers And Public Work + +Private workers are account-restricted. Public cross-account checks require a +separate signed and SSRF-safe grant. Empty account scope continues to mean a +platform-operated worker and must never be customer-selectable. + +### Raft Maturity + +Membership, persistence, snapshots, transport, and placeholder FSM entries +exist. Production `distributed_critical` behavior does not. The control plane +is an external witness, never a voter; provider calls remain outside Raft. + +## Maintenance + +When a new RSMon plan adds worker behavior, update this ledger and one concrete +implementation document in the same change. Do not copy a planning draft into +this repository without reconciling it against current packages, wire types, +security boundaries, and tests. diff --git a/docs/tasks-and-notifications.md b/docs/tasks-and-notifications.md new file mode 100644 index 0000000..d935eae --- /dev/null +++ b/docs/tasks-and-notifications.md @@ -0,0 +1,131 @@ +# Normal Tasks And Notifications + +## Normal Check Execution + +Normal work is centrally scheduled. The control plane owns PostgreSQL task +rows, selection, leases, retries, deadlines, and dead-letter state. The worker +owns only one execution attempt and its terminal report. + +The implemented path is: + +1. `internal/distworker/client.go` opens the authenticated WebSocket. +2. `internal/distworker/runner.go` selects a recognized populated task branch. +3. A bounded worker pool executes the task under its deadline. +4. `internal/checkexec/exec.go` dispatches to a database-free check package. +5. The runner queues one result and the result writer returns it to the control + plane with the exact lease token. + +Implemented check kinds are HTTP, SSL, SSH, FTP, DNS, WHOIS, BSSL, LLM, +LLM-HTTP, ping, TCP, and UDP. Control-plane selection uses capabilities. The +local dispatcher supports only its explicit switch cases, but strict malformed +envelope and unsupported-kind result handling remains work package N1 rather +than a complete fail-closed protocol response. + +## Queue And Shutdown Rules + +- Input and output queues are bounded. Backpressure must not create unbounded + goroutines or memory use. +- Concurrency is supplied by the control plane but clamped by the local + maximum. +- A task panic is recovered at the task boundary and reported as a failed + attempt; it must not kill the runner. +- SIGTERM stops accepting tasks, allows bounded in-flight completion, attempts + final result delivery, and then exits. +- WebSocket reconnect does not re-run an in-flight or completed task. +- HTTP polling remains compatibility-only and must not become a second normal + scheduler. + +## Delegated Notifications + +`internal/distworker/notification.go` executes one pre-rendered delivery +attempt. It does not render alert policy and does not own retry scheduling. + +| Method | State | Credential source | +| --- | --- | --- | +| Email | Implemented | scoped SMTP list from init/config | +| Telegram | Implemented | scoped bot list from init/config | +| Webhook | Implemented | current shared signing configuration | +| Mattermost | Implemented | current shared defaults/task endpoint | +| SMS | Explicitly unsupported | no provider selected | +| Voice | Explicitly unsupported | no provider selected | + +SMS and voice return `permanent` with `unsupported_method`; they must never be +reported as delivered or silently dropped. + +Worker result statuses are `delivered`, `retryable`, `permanent`, and +`partial`. Transport failures, provider rate limits, and provider 5xx responses +are retryable. Invalid recipients, authentication failures, malformed payloads, +and unsupported methods are permanent. Positive provider retry-after values +are included in the result; the control plane owns the actual retry time. + +## Credentials + +Credentials arrive in `WorkerInit.Credentials` and remain in memory. SMTP and +Telegram tasks may request a credential ID; the worker resolves only that ID +from the authorized pushed set. Missing IDs are permanent failures. Do not +fall back to another account's or platform credential. + +The following fields must be redacted from logs and UI buffers: password, +token, API key, signing secret, hook URL, authorization headers, and fields +ending in `_secret`. Provider response text is bounded and must not include +response bodies that could contain secrets. + +Webhook and Mattermost need persisted per-account credentials in the control +plane before they can claim the same isolation guarantees as SMTP and +Telegram. The worker wire shape already separates method configuration; do not +introduce environment-global fallback for private workers. + +## Selfcheck Notifications + +System selfcheck alerts are not normal notification tasks. They allow a worker +to notify configured system contacts when the control plane itself is +unreachable. They use system contacts and credentials from config, local +jitter, and local deduplication. They must not impersonate customer delivery or +mutate control-plane message state. + +The local notifications page currently records selfcheck deliveries. Normal +delegated delivery attempts should be added to the same bounded view after +redaction, with job ID, method, status, duration, and time only. + +## Implementation Work Packages + +### N1: Runner Integration Coverage + +Add strict envelope/inner job-ID validation and structured unsupported-kind +results. Add WebSocket integration tests for reconnect, task panic recovery, +queue backpressure, result resend, stale lease behavior, malformed envelopes, +unsupported kinds, and graceful drain. Target +`internal/distworker/runner_protocol_test.go` and a local test WebSocket server. + +### N2: Complete Local Delivery Audit + +Record delegated notification outcomes in the bounded local notification ring. +Never store recipient values, rendered body, credentials, or full provider +responses in SQLite. Verify `/notifications` shows both selfcheck and delegated +attempts without leaking secrets. + +### N3: Per-Account Webhook And Mattermost Credentials + +Once the control plane sends credential IDs and account-scoped records, require +an exact credential match just like SMTP and Telegram. Remove global fallback +for private workers. Add cross-account negative tests. + +### N4: SMS And Voice Providers + +Select a provider and define a credential wire type before adding an executor. +Implementation must include timeout, retry classification, provider response +redaction, idempotency support, and contract tests. Until all are present, +retain the explicit permanent unsupported result. + +## Acceptance Tests + +- Every supported check kind can execute from a serialized task without a + database connection. +- A stale or missing lease token cannot produce an accepted normal result. +- One task causes at most one local execution during reconnect and duplicate + frame delivery. +- Private workers cannot resolve a credential outside their authorized set. +- SMTP 4xx/rate-limit conditions are retryable; invalid authentication is + permanent. +- SMS and voice remain explicit failures until provider tests exist. +- Logs and local UI contain no pushed credential values. diff --git a/docs/web-console-and-observability.md b/docs/web-console-and-observability.md new file mode 100644 index 0000000..7cbb110 --- /dev/null +++ b/docs/web-console-and-observability.md @@ -0,0 +1,128 @@ +# Web Console And Host Observability + +## Current Implementation + +`internal/webapp` is a functional embedded HTTP application backed by SQLite. +It provides authenticated overview, apps, checks, notifications, logs, status, +settings, updates, password rotation, token rotation, audit retention, and +cluster status endpoints. Static assets and templates are embedded in the +worker binary. + +The listener defaults to `WORKER_HOST=0.0.0.0` and `WORKER_PORT=27401`. +Compose publishes it on host loopback by default through `WORKER_BIND_IP`; that +variable is a Compose interpolation setting, not a variable read by the binary. + +## Authentication + +Two modes are implemented: + +| Configuration | Behavior | +| --- | --- | +| Both `WORKER_LOGIN` and `WORKER_PASSWORD` set | Basic-auth-backed operator sessions and `/web/api/*` access | +| Both empty | Local bcrypt user; a one-time password is printed on first start | +| Only one set | Startup error | + +Sessions are HTTP-only and SameSite strict, with a 30-minute idle and 8-hour +absolute lifetime. State-changing browser requests require CSRF validation. +Authenticated responses use no-store and restrictive security headers. + +OAuth/device authorization is not implemented. Public exposure requires a TLS +reverse proxy and network restrictions. Do not claim that `WORKER_URL` enables +TLS; it only advertises the externally reachable URL. + +## Known Runtime Gap + +`webapp.New` constructs `Inventory` and `Metrics`, but `Server.Start` currently +does not start either collector and `Server.Close` does not stop them. As a +result, production `/apps` and `/status` pages can remain empty even though +collector unit tests pass. + +This is the first required web-console change: + +1. Start both collectors with the server context before accepting requests. +2. Stop and wait for them during `Close` and context cancellation. +3. Make collector start idempotent and collector failure observable. +4. Add a server integration test that observes populated inventory and metrics + after startup. + +## Inventory View + +The current local collector reads Linux `/proc`, process command lines and +working directories, TCP listeners, and resource data, then stores snapshots +in SQLite. It does not yet provide complete grouping, Docker, Compose, systemd, +nginx, UDP, or deploymentd-compatible inventory. The implementation plan is in +[inventory.md](inventory.md). + +## Host Metrics + +There are two related collectors: + +- `internal/distworker/server_metrics.go` sends the linked server's CPU, + memory, disk, load, uptime, process, and network snapshot to the control + plane every five seconds. This path is implemented. +- `internal/webapp/metrics.go` drives local status pages and a local ring. It + exists but is affected by the lifecycle gap above. + +The control-plane report uses interval rates for CPU and aggregate networking. +The local metrics implementation currently computes CPU from lifetime totals; +align it with interval deltas so local and remote views agree. + +## Compose Management Target + +Compose management remains planned and is opt-in. Implement it after read-only +inventory is complete, under a separate `internal/composeops` package. + +Required constraints: + +- fixed stacks root, default `/opt/stacks`, with canonical path containment; +- explicit `docker compose -f ` invocation; +- no shell interpolation and no free-form host command endpoint; +- per-stack single-flight action lock; +- async `202 Accepted` operations with bounded progress streaming; +- viewer/operator/admin permissions and audit records; +- typed confirmation for down, delete, and restore; +- compose validation before save and backup before deploy; +- Docker socket treated as root-equivalent and disabled by default. + +The first Compose release includes read/list/status/logs only. Mutation, file +editing, environment editing, terminal access, backup, and restore are separate +release gates rather than one large feature switch. + +## Host Status Extensions + +Implement in this order: + +1. Correct `/proc`, `statfs`, and `/proc/net/dev` interval collection. +2. Add bounded local 24-hour one-minute aggregates. +3. Add optional `lsblk --json` device topology. +4. Add optional cached SMART and sensors data. +5. Add Docker daemon and per-container status only when the socket capability + is enabled. + +Missing commands or permissions produce unavailable sections, not worker +startup failure. All subprocesses use fixed argument arrays, deadlines, output +limits, and allowlisted executable names. + +## Health And Updates + +`GET /healthz` is local process liveness. The image healthcheck should call +this endpoint, not `rsmon-worker health`, because the latter probes the control +plane `/up` endpoint. Control-plane outage is a selfcheck/readiness event, not +proof that the worker process is dead. + +In-place binary replacement from the web UI is not implemented and should not +run with ambient root access. The systemd update flow should download a signed +artifact to a staging path and require an external privileged installer or +explicit operator command to activate it. + +## Acceptance Tests + +- Starting `webapp.Server` produces nonempty metrics and process inventory on + Linux and stops every collector cleanly. +- Both auth modes work, XOR credentials fail startup, CSRF protects writes, + and unauthenticated API calls fail. +- A public bind is documented as insecure without TLS; no credential is logged. +- `/healthz` remains healthy during a simulated control-plane outage. +- Every host command has timeout/output-limit tests and rejects user-supplied + executable or path traversal. +- Compose mutation cannot operate outside the configured stacks root.