docs: migrate worker implementation plans
Все проверки выполнены успешно
CI / test (push) Successful in 30s
Docker / Build and publish worker image (push) Successful in 10m49s

Этот коммит содержится в:
Gleb Tv
2026-07-13 18:19:00 +03:00
родитель 16922d5081
Коммит fd1a010e31
12 изменённых файлов: 1506 добавлений и 4 удалений

Просмотреть файл

@@ -57,7 +57,12 @@ Published images use these tags:
- `sha-<12-character-commit>` for every push; - `sha-<12-character-commit>` for every push;
- `latest` for `master`; - `latest` for `master`;
- the exact `v*` tag for releases. - the `v*` release ref, with Docker-invalid characters replaced by `-`.
The Gitea workflow reads `HARBOR_REGISTRY`, `HARBOR_USER`, and
`HARBOR_PASSWORD`. `HARBOR_REGISTRY` may be a host such as `reg.rsxx.ru` or an
HTTP(S) URL; the workflow strips the scheme and trailing slash before composing
Docker image references. The Harbor project is appended separately as `rsmon`.
## systemd ## systemd
@@ -113,9 +118,17 @@ has only `CAP_NET_RAW` for ICMP checks.
| `WORKER_CLUSTER_PEERS` | no | none | Comma-separated `node@host:port` peers. | | `WORKER_CLUSTER_PEERS` | no | none | Comma-separated `node@host:port` peers. |
| `WORKER_CLUSTER_DATA_DIR` | with cluster | none | Persistent Raft state directory. | | `WORKER_CLUSTER_DATA_DIR` | with cluster | none | Persistent Raft state directory. |
The public liveness endpoint is `GET /healthz`. `rsmon-worker health` checks the The public liveness endpoint is `GET /healthz`; the image probes it with
configured control plane's `/up` endpoint and is suitable for container health `rsmon-worker liveness`. `rsmon-worker health` separately checks the configured
checks. control plane's `/up` endpoint for connectivity diagnostics.
## Implementation Documentation
Worker architecture, protocol, inventory, private-worker isolation, network
diagnostics, web console, and critical-cluster work are specified in
[`docs/README.md`](docs/README.md). These documents replace worker-owned planning
material formerly kept in the RSMon control-plane repository and include a
source migration ledger.
## Security ## Security

72
docs/README.md Обычный файл
Просмотреть файл

@@ -0,0 +1,72 @@
# Worker Implementation Documents
This directory is the implementation authority for the standalone RSMon worker.
It converts the worker-related plans formerly kept in the RSMon control-plane
repository into contracts tied to this repository's packages and tests.
These documents distinguish three states:
- **Implemented**: wired into `cmd/rsmon-worker` and covered by tests.
- **Partial**: code exists, but a named runtime path or acceptance criterion is
missing.
- **Planned**: no production path exists yet. The document identifies the
package, protocol, dependency, and tests required to implement it.
Plans do not override code. When a document and the current implementation
disagree, update both in the same change or mark the discrepancy explicitly.
## Implementation Index
| Document | Scope | Current state |
| --- | --- | --- |
| [architecture.md](architecture.md) | Process boundaries, ownership, runtime composition | Implemented with partial subsystems |
| [control-plane-protocol.md](control-plane-protocol.md) | WebSocket frames, leases, config, compatibility APIs | Implemented |
| [tasks-and-notifications.md](tasks-and-notifications.md) | Normal checks and delegated notification execution | Implemented with provider gaps |
| [web-console-and-observability.md](web-console-and-observability.md) | Local UI, auth, inventory view, host telemetry | Partial |
| [inventory.md](inventory.md) | Worker host discovery and control-plane inventory projection | Partial |
| [private-workers.md](private-workers.md) | Customer worker isolation, credentials, onboarding | Partial |
| [network-diagnostics.md](network-diagnostics.md) | Confirmation checks and dedicated diagnostic tasks | Partial |
| [critical-check-cluster.md](critical-check-cluster.md) | Raft-backed dispatchless critical checks | Scaffold only |
| [implementation-roadmap.md](implementation-roadmap.md) | Ordered repository work packages and release gates | Active |
| [source-plan-migration.md](source-plan-migration.md) | Source-to-target conversion ledger and resolved conflicts | Complete mapping |
## Ownership Boundary
This repository owns:
- worker process startup, configuration, and graceful shutdown;
- the control-plane client and wire types;
- check and notification execution;
- local web console, local state, inventory collection, and host metrics;
- worker-to-worker transport and Raft state;
- Docker, Compose, systemd, and release-image packaging.
The RSMon control-plane repository owns:
- users, accounts, monitors, contacts, billing, and RBAC;
- PostgreSQL task production, selection, leasing, retry, and dead-letter state;
- worker registration and authorization policy;
- accepted-result application and VictoriaMetrics persistence;
- customer-facing fleet, inventory, diagnostics, and incident APIs.
The worker must not receive control-plane PostgreSQL, Valkey, or
VictoriaMetrics credentials. Normal results, host telemetry, inventory, and
diagnostics are reported over authenticated protocol messages for validation
and persistence by the control plane.
## Required Quality Gates
Run before merging worker changes:
```bash
make check
docker build -t rsmon-worker:test .
docker run --rm rsmon-worker:test --version
docker compose config
```
Features that alter protocol messages require compatibility tests in
`internal/wire` and runner protocol tests in `internal/distworker`. Raft work
requires deterministic FSM tests and a multi-node fault test. Security-sensitive
features require negative tests for account scope, arbitrary target rejection,
credential redaction, and unauthenticated access.

85
docs/architecture.md Обычный файл
Просмотреть файл

@@ -0,0 +1,85 @@
# Worker Architecture
## Runtime
`cmd/rsmon-worker/main.go` composes three independently stoppable subsystems:
1. `internal/distworker`: persistent control-plane connection, normal task
execution, selfcheck, notification delivery, and server metric reports.
2. `internal/webapp`: authenticated local operator console and SQLite-backed
local state.
3. `internal/workercluster`: optional worker-to-worker Raft transport and FSM.
All three share one cancellation context. The worker can run without the web
console using `--no-web`; the cluster is disabled unless
`WORKER_CLUSTER_ENABLED=true`.
## Two Execution Paths
The two check paths are deliberately separate.
| Path | Work origin | Ownership | Worker implementation |
| --- | --- | --- | --- |
| Normal checks and notifications | Control-plane PostgreSQL `tasks` rows | Exact lease token and expiry held by control plane | `internal/distworker` |
| `distributed_critical` | Signed config adopted by a worker Raft cluster | Raft FSM incident and outbox metadata | `internal/workercluster` plus a planned scheduler bridge |
Normal work always uses control-plane dispatch and leases. The critical path
must never use a normal task row, job assignment, or lease. Adding the critical
path must not change normal protocol behavior.
## Package Boundaries
| Package | Responsibility | Must not do |
| --- | --- | --- |
| `internal/wire` | JSON-compatible protocol types | Network or persistence work |
| `internal/distworker` | Connections, queues, pools, execution orchestration | Direct control-plane database writes |
| `internal/checkexec` | Database-free check dispatch | Scheduling or result persistence |
| `checks/*` | Protocol-specific probes | Worker selection or account policy |
| `internal/webapp` | Local UI, sessions, audit, local snapshots | Become the fleet source of truth |
| `internal/workercluster` | Raft transport, log, FSM, snapshots, membership | Store raw samples or plaintext credentials |
| `internal/sender` | Provider calls shared by worker executors | Own retries or durable task state |
New host discovery should move into `internal/inventory`; the web console may
read it through an interface but must not remain the collector's owner. New
network probes should live in `internal/netdiag` and use an allowlisted target
provided by a validated control-plane task.
## Current Status
| Capability | State | Evidence |
| --- | --- | --- |
| WebSocket normal check execution | Implemented | `internal/distworker/client.go`, `runner.go` |
| Check kinds | Implemented | `internal/checkexec/exec.go`, `checks/*` |
| Delegated notifications | Partial | email, Telegram, webhook, Mattermost implemented; SMS/voice unsupported |
| Host telemetry to control plane | Implemented | `internal/distworker/server_metrics.go` |
| Local web console | Implemented | `internal/webapp/server.go`, `routes.go` |
| Local inventory and status pages | Partial | collectors exist but are not started by `Server.Start` |
| Private-worker scheduler isolation | Implemented in shared selector code | `app/models/worker_node.go`, `task_selector.go`, `check_jobs.go` |
| Private-worker bootstrap and worker-side scope verification | Planned | no one-time bootstrap or signed config path |
| Cross-worker confirmation | Partial | peer selfcheck and shared model logic exist; no rich diagnostic tasks |
| Raft membership and persistence | Partial | `internal/workercluster` |
| Dispatchless critical checks | Planned | no scheduler, signature adoption, quorum incident engine, or outbox executor |
## Data Rules
- Credentials delivered in `init`/`config` stay in memory and are never written
to the local SQLite store, Raft log, or snapshots.
- Raw check samples and high-volume telemetry never enter Raft.
- Local UI state is not authoritative control-plane state.
- A normal result is not valid without the current exact lease token.
- Any worker-originated account, server, monitor, or target identifier is
untrusted until the control plane validates it against the authenticated
worker.
- A worker must not execute a synthetic health check whose target is itself.
## Runtime Completion Criteria
- SIGTERM drains the runner, stops collectors, closes listeners, shuts down
Raft, and closes SQLite without exceeding the service stop timeout.
- A web-console failure is reported and causes an intentional process policy;
it must not silently leave a partially healthy process.
- Startup rejects malformed `WORKER_URL`, incomplete auth credentials, invalid
cluster settings, and unwritable data directories before accepting work.
- `GET /healthz` reports process-local liveness. Control-plane reachability is
a separate readiness/selfcheck signal and must not make a healthy container
fail its local liveness probe.

156
docs/control-plane-protocol.md Обычный файл
Просмотреть файл

@@ -0,0 +1,156 @@
# Control-Plane Protocol
## Authority
`internal/wire/types.go` is the executable schema. New fields must be optional
during rollout, and new branches require decode tests against both old and new
frames. Version strings are informational; explicit capabilities select
protocol features.
The worker connects to:
```text
GET /worker?token=<RSMON_TOKEN>
```
`/api/worker` and the HTTP jobs/results APIs remain compatibility paths. New
workers use WebSocket task envelopes.
## Frame Model
Every frame is a `wire.WorkerMessage` with `kind` and one active content
branch.
| Direction | `kind` | Active branch | State |
| --- | --- | --- | --- |
| Server to worker | `init` or `config` | `init` | Implemented |
| Server to worker | `task` | `task_envelope` | Implemented |
| Worker to server | `result` | `result` | Implemented |
| Worker to server | `result` | `notification_result` | Implemented |
| Worker to server | `result` | `server_metric` | Implemented |
| Worker to server | `heartbeat` | `heartbeat` | Implemented |
| Either | `error` | `error` | Implemented |
| Worker to server | `result` | inventory report | Planned |
| Server to worker | cluster config/witness | dedicated typed branch | Planned |
| Server to worker | diagnostic task | new task-envelope variant | Planned |
The legacy top-level `task` and `notification_task` branches are accepted for
rollout compatibility. If a frame includes both a valid current
`task_envelope` and a legacy branch, the worker executes only
`task_envelope`. The current runner ignores malformed or unsupported task
branches and does not yet compare the envelope job ID with the inner job ID;
strict rejection and reporting are P1 work below.
## Initialization And Refresh
`wire.WorkerInit` supplies runtime values owned by the control plane:
- worker ID, region, advertised URL, capabilities, and concurrency;
- allowed notification methods and account IDs;
- optional linked server ID for host metrics;
- LLM endpoints;
- scoped notification credentials and system contacts;
- peer workers for selfcheck/cluster-adjacent behavior.
The worker clamps supplied concurrency to its local maximum. Credentials are
replaced atomically in memory on refresh. Removed credentials must become
unavailable immediately after the refresh is applied.
Private-worker hardening will add an immutable worker account ID, config
version, expiry, and signature. Until then the executable trusts the
authenticated control plane to send a correctly scoped config; server-side
selection remains the primary isolation boundary.
## Normal Task Envelope
`wire.TaskEnvelope` is a tagged union:
```json
{
"kind": "task",
"task_envelope": {
"type": "check",
"job_id": "uuid",
"check": {
"job_id": "uuid",
"lease_token": "per-lease-secret",
"check_id": 123,
"monitor_id": 456,
"kind": "http",
"host": "example.com",
"url": "https://example.com",
"interval": 60,
"settings": {}
}
}
}
```
`type=notification` activates `notification` instead. Job IDs in the envelope
and active branch must match once P1 validation lands. Today a task without a
recognized populated branch is ignored without execution.
## Result Invariants
- Echo `job_id` and the exact `lease_token` from the task.
- Send one terminal result per execution attempt.
- Never retry a result by executing the task again. Result transport retries
resend the same terminal report.
- Treat duplicate terminal acknowledgements as success.
- Do not infer task acceptance from a WebSocket write alone; durable ownership
remains on the control plane until it validates the result.
- Bound error strings and provider responses before transmission.
## Host Metrics
`wire.ServerMetricReport` is sent only after `WorkerInit.ServerID` is present.
The worker collects locally and sends bounded snapshots. The control plane
validates worker/server/account ownership and persists both the latest cache
and VictoriaMetrics points. The worker does not have TSDB credentials.
## Protocol Work Packages
### P1: Conformance Tests
Files:
- `internal/wire/types_test.go`
- `internal/distworker/runner_protocol_test.go`
Tests:
- decode every current frame branch;
- prefer the current envelope over duplicate legacy branches;
- reject mismatched envelope and inner job IDs;
- preserve unknown optional fields during compatible rollout;
- reject missing lease tokens before execution result submission;
- atomically replace config and credentials.
### P2: Safe Token Rotation
Current `Runner.RotateToken` closes the runner stop channel. Replace this with a
connection-scoped cancellation path that stores the new token, closes only the
active WebSocket, and reconnects without terminating worker services.
Acceptance: rotating from the web console produces a reconnect using the new
token while the HTTP listener, collectors, and optional cluster stay running.
### P3: Signed Private-Worker Config
Add to the init/config branch:
- `account_id` for private workers;
- monotonic `config_version`;
- `issued_at` and `expires_at`;
- signature key ID and Ed25519 signature over canonical payload bytes.
Reject regressions, invalid signatures, expired config, and account changes.
Keep the last valid config only until its expiry; do not silently accept an
invalid replacement.
### P4: New Typed Branches
Inventory, diagnostic, and critical-cluster messages each receive a dedicated
wire type. Do not tunnel them through `event` or arbitrary `json.RawMessage`.
Each branch must define payload limits, account/target validation ownership,
idempotency, and compatibility behavior before implementation.

188
docs/critical-check-cluster.md Обычный файл
Просмотреть файл

@@ -0,0 +1,188 @@
# Dispatchless Critical-Check Cluster
## Status
`internal/workercluster` is a scaffold, not a production critical-check engine.
It currently provides Hashicorp Raft, bbolt log/stable storage, snapshots,
HTTP-based authenticated transport, bootstrap/join/membership operations, log
entry types, a small FSM, and operator status.
It does not yet execute `distributed_critical` checks, verify signed config,
evaluate observation/region/notification quorum, encrypt snapshots, deliver a
commit-backed outbox, or consume an external witness report.
The hardcoded test-config endpoint and startup flag are development-only and
must stay disabled in production.
## Non-Negotiable Separation
`distributed_critical` is a distinct path:
- no control-plane job dispatch;
- no PostgreSQL task row;
- no task lease or single worker owner;
- every eligible observer executes the adopted check deterministically;
- raw samples go to the control plane/TSDB outside Raft;
- classified observations, compact incidents, and outbox metadata enter Raft;
- normal checks and normal notifications continue unchanged.
The control plane signs configuration, receives replay, and acts as an external
witness. It is never a Raft voter.
## Topology
- Production clusters have 3 or 5 voters, never an even count.
- Voters remain within a bounded-latency topology. Remote regions use observer
nodes unless measured RTT supports the configured election timeout.
- Nodes may be `voter`, `observer`, or `voter+observer`.
- Observer count affects observation policy, not Raft election quorum.
- A node without durable, real-fsync storage cannot be a voter.
Single-voter bootstrap is temporary. The cluster must not execute customer
critical checks until at least three voters are healthy and the signed
observer set is committed.
## Raft State
The FSM contains only:
- monotonic config version and adopted critical-check definitions;
- versioned observer set;
- compact incident state and committed observations;
- bounded notification outbox metadata and idempotency tombstones;
- compact member diagnostics and partition/witness state;
- tenant isolation policy and applied-version index.
It never contains raw probe samples, response bodies, provider bodies,
plaintext credentials, credential envelopes, or normal task leases.
`raft-boltdb/v2` is only the local Raft `LogStore` and `StableStore`; it is not
the application model. Snapshots are produced by the FSM and must be versioned,
checksummed, and encrypted at rest before production use.
## Log Entries
Allowed application entry kinds:
- `config.adopt`
- `observer_set.update`
- `incident.observe`
- `incident.transition`
- `outbox.enqueue`
- `outbox.delivered`
- `outbox.ack`
- `partition.report`
- `membership.propose_add`
- `membership.demote`
- `membership.remove`
- `diagnostics.update`
No `task.lease.*`, retry, completion, or dead-letter entries belong in this
package.
## Signed Config Adoption
The control plane sends an Ed25519-signed canonical payload containing:
- cluster and tenant IDs;
- monotonic config version and expiry;
- critical checks and incident policies;
- proposed observer set, role/region map, and content hash;
- signing key ID and credential-envelope references.
Every node verifies signature, cluster identity, expiry, monotonic version,
observer-set hash, and supported schema. The leader proposes `config.adopt` and
`observer_set.update`. No observer executes the new config until both commits
are applied locally. Invalid config is rejected and reported through compact
diagnostics; there is no silent downgrade.
## Scheduling And Observation
For each adopted check, eligible observers derive the same interval boundary
from `epoch + n*interval`. Per-worker jitter is deterministic from worker ID,
check ID, observer-set version, and interval number. At each tick:
1. Run the underlying probe through `internal/checkexec` outside Raft.
2. Send raw metrics through the normal control-plane metrics path.
3. Classify to `ok`, `warn`, `down`, or `unknown` using signed policy.
4. Propose `incident.observe` to the leader with committed timestamp and
observer/config versions.
Followers forward proposals or return a typed not-leader response containing
the current leader. They never silently drop observations.
## Four Quorums
- `raft_quorum`: majority of voters required to commit.
- `observation_quorum`: observers agreeing within the observation window.
- `region_quorum`: represented regions agreeing on state.
- `notification_quorum`: region agreement required for first customer alert.
All names remain distinct in code, metrics, configuration, and logs.
## Deterministic Incident FSM
`FSM.Apply` must perform no I/O, network calls, wall-clock reads, randomness,
or provider calls. It uses only committed payload values and prior state.
Incident lifecycle is `clear -> observing -> open -> resolving -> clear`.
Policy includes confirmation count, observation window, classification,
minimum incident dwell, transition-rate suppression, cooldown, and outbox retry
metadata. Apply enforces bounded state and deterministic idempotency keys.
The current FSM merely records the latest observation and accepts externally
constructed transitions. Replace this placeholder with deterministic policy
evaluation and tests before running real checks.
## Commit-Before-Notify
Strict order:
1. Probe and classify outside Raft.
2. Commit observation at `raft_quorum`.
3. FSM deterministically commits incident transition and outbox metadata.
4. After the committed outbox entry is visible, an executor calls the provider
outside Raft.
5. Provider acknowledgement produces `outbox.delivered`; retry/failure produces
bounded metadata updates.
Provider credentials stay in node-local memory/secure storage and are never
part of an entry or snapshot. Delivery uses stable channel idempotency keys to
survive leader failover.
## Partitions And Witness
Partition states are `steady`, `degraded`, `partitioned`, `healing`,
`split_brain_detected`, and `witness_only`. Loss of `raft_quorum` freezes
incident transitions and outbox creation. A cluster must never fabricate an
open or recovery while partitioned.
The external witness reports observed leader/term, reachable voters and
observers, split-brain flag/time, and report time. It can alert and inform state
but cannot commit or fabricate incidents.
## Ordered Implementation
1. Remove production exposure of debug config application.
2. Complete versioned FSM types, command validation, and deterministic tests.
3. Add mTLS identity and safe one-claim cluster bootstrap.
4. Add signed config and observer-set adoption.
5. Add deterministic scheduler and checkexec bridge in shadow mode.
6. Implement observation aggregation, incident policy, and idempotency.
7. Implement encrypted snapshots and restore/migration tests.
8. Add metadata outbox executor and failover-safe delivery.
9. Add external witness, replay, metrics, and operational runbooks.
10. Run synthetic 3/5-node fault campaigns before any customer check.
## Release Gates
- Two independent bootstraps for one cluster are detected and rejected.
- Changes that leave fewer than three or an even number of voters are rejected.
- Leader failover completes within twice the configured election timeout.
- A minority partition commits zero incident transitions.
- Loss of observation quorum with intact Raft quorum sends no notification.
- Snapshot inspection finds no raw samples or credential material.
- Signature mutation and observer-set version mismatch prevent execution.
- Killing the leader between provider acknowledgement and metadata commit does
not create duplicate customer-visible notification effects.
- Normal WebSocket tasks and notifications continue throughout cluster tests.

165
docs/implementation-roadmap.md Обычный файл
Просмотреть файл

@@ -0,0 +1,165 @@
# Worker Implementation Roadmap
This roadmap orders work by dependency and risk. A package existing does not
make a work package complete; its release gate must pass in a wired process.
## R0: Distribution Reliability
State: active.
Worker repository:
- publish valid Harbor references after normalizing a configured registry URL;
- use local `/healthz` for container liveness;
- verify amd64 and arm64 image startup, Chromium availability, and version
metadata;
- add a package/install smoke test for Docker and systemd artifacts;
- document immutable SHA and release tags as production defaults.
Gate: a push publishes `sha-<12>` and `latest` manifests for both platforms,
and a container remains healthy when the control plane is unavailable.
## R1: Runtime Correctness
State: partial.
Worker repository:
- wire local inventory and metrics collector lifecycle into web server start
and shutdown;
- validate full HTTP config, including `WORKER_URL`, in main startup;
- make token rotation reconnect rather than stop the runner;
- define process policy when the web listener exits unexpectedly;
- add WebSocket reconnect, drain, duplicate frame, and backpressure tests.
Gate: collectors populate real pages, rotation preserves all subsystems, and
SIGTERM leaves no listener, collector, task, or SQLite goroutine behind.
## R2: Protocol And Credential Hardening
State: partial.
Worker repository:
- complete frame conformance tests;
- add signed config version/account identity;
- enforce task account and credential scope locally;
- add bounded delegated-delivery audit to the local console;
- support per-account webhook/Mattermost credentials after control-plane wire
support exists.
Control-plane dependency:
- signed config producer and key rotation;
- persisted account-scoped webhook/Mattermost credentials;
- lease/task account fields treated as immutable during result application.
Gate: cross-account fixtures fail before network execution and no credential
appears in worker logs, SQLite, snapshots, or protocol error payloads.
## R3: Private-Worker Lifecycle
State: planned beyond scheduler isolation.
Worker repository:
- one-time bootstrap exchange and atomic secret storage;
- token handoff and revocation handling;
- stale/expired signed-config behavior;
- clear disabled/revoked UI state;
- optional mTLS client identity.
Control-plane dependency:
- customer worker CRUD and billing entitlement;
- one-time bootstrap token state;
- immutable account binding and signed config;
- audit and revocation acknowledgement.
Gate: a customer can install, connect, execute only their work, rotate, disable,
revoke, and uninstall without an operator admin secret.
## R4: Host Inventory And Console
State: partial.
Worker repository:
- extract `internal/inventory` and start process/host collection;
- add typed bounded inventory report;
- add Docker/Compose, nginx, systemd, and address collectors in stages;
- add read-only Compose status/logs before mutation;
- correct local interval metrics and add optional hardware sections.
Control-plane dependency:
- worker inventory report ingestion and source-aware reconciliation;
- server/worker/account ownership validation;
- deploymentd comparison and monitor suggestion UI.
Gate: partial failures cannot delete inventory, worker and deploymentd sources
coexist, and reports contain no environment or credential values.
## R5: Dedicated Network Diagnostics
State: planned; normal confirmation is already partial.
Worker repository:
- add diagnostic wire branch and `internal/netdiag`;
- ship HTTP GET/HEAD and unauthenticated SSH handshake first;
- add TCP/DNS, then separately gate traceroute/MTR;
- implement SSRF, redirect, timeout, output, and rate protections.
Control-plane dependency:
- monitor-bound target normalization and signed target hash;
- diagnostic task production, storage, RBAC, rate limits, and UI;
- private-worker source eligibility.
Gate: no task can probe outside its authorized monitor target and all results
are bounded, structured, and credential-free.
## R6: Public Check Economy
State: planned.
Depends on R2, R3, and diagnostic-grade target protection. Public work uses a
separate signed grant and safe HTTP methods, never widened account scope.
Gate: private targets and credential-bearing requests are impossible under DNS
rebinding and redirects; accounting remains idempotent under result replay.
## R7: Critical-Check Cluster
State: Raft scaffold only.
Worker repository:
- secure bootstrap and mTLS identity;
- signed config adoption and observer-set versioning;
- deterministic scheduler and check executor bridge;
- deterministic incident/quorum FSM;
- encrypted snapshots and restore;
- metadata outbox, witness, replay, and metrics;
- 3/5-node fault campaigns.
Control-plane dependency:
- critical-check config/signing service;
- credential-envelope service;
- witness and replay endpoints;
- customer incident projection and audit.
Gate: all release gates in
[critical-check-cluster.md](critical-check-cluster.md) pass while normal tasks
continue without behavioral change.
## Documentation Rule
Every completed work package updates:
- the relevant implementation document's current-state section;
- this roadmap state and gate evidence;
- root configuration examples when environment or deployment changes;
- the source migration ledger if a control-plane plan is superseded.

165
docs/inventory.md Обычный файл
Просмотреть файл

@@ -0,0 +1,165 @@
# Worker Host Inventory
## Purpose And Boundary
Inventory connects checks to the host, service, Compose project, domain, and
deployment that they observe. The worker collects facts; the RSMon control
plane validates identity, applies account scope, reconciles lifecycle, and
persists the inventory projection.
The control-plane inventory entities remain `Server`, `ServerIp`, `Site`,
`Deployment`, `Domain`, `Repo`, and `site_repos`. Their API and rstuff stream
projection are control-plane concerns. This repository owns only local
discovery and worker-originated reports.
## Existing Collectors
Current state:
- `internal/webapp/inventory.go` discovers Linux processes, command lines,
working directories, TCP listeners, and basic resource facts.
- snapshots are stored in the local web-console SQLite database;
- `internal/distworker/server_metrics.go` reports bounded process/network host
snapshots, but not normalized application inventory;
- external `deploymentd` remains the implemented authoritative collector for
nginx, Docker Compose, and host inventory on the control plane.
The worker does not currently send an inventory report to the control plane.
It must not claim deploymentd parity until the protocol, ingestion, and
reconciliation tests below are complete.
## Target Package Layout
Move collection ownership out of `internal/webapp`:
```text
internal/inventory/
collector.go orchestration and partial-success envelope
process_linux.go procfs process and listener discovery
docker.go Docker and Compose discovery behind capability
nginx.go read-only nginx virtual-host discovery
systemd.go allowlisted unit discovery
normalize.go stable IDs and control-plane report conversion
store.go optional local snapshot interface
```
The web console consumes a read-only snapshot interface. The runner consumes a
bounded report interface. Collectors return per-section errors; one failed
section does not discard successful sections.
## Report Contract
Add a typed `WorkerInventoryReport` branch to `internal/wire`:
```json
{
"kind": "result",
"inventory": {
"schema": "rsmon.worker.inventory.v1",
"report_id": "uuid",
"server_id": 42,
"collected_at": "2026-07-13T15:00:00Z",
"full_snapshot": true,
"host": {},
"addresses": [],
"processes": [],
"compose_projects": [],
"nginx_sites": [],
"systemd_units": [],
"errors": [{"section": "docker", "code": "unavailable"}]
}
}
```
Constraints:
- `report_id` is stable across transport retries and unique across collection
runs.
- `server_id` comes from authenticated `WorkerInit.ServerID`, never local env.
- A full snapshot permits missing-item reconciliation; partial reports do not.
- Section and total item counts, strings, labels, and serialized bytes are
bounded before enqueue.
- Environment variable values, file contents, process environments, container
secrets, and Compose `.env` values are never included.
- The control plane rejects a report when the worker is not linked to the
server or the private worker's account does not own it.
## Normalization
Stable local identity keys:
- process app: executable plus canonical working directory;
- Compose site: Docker Compose project name plus canonical project directory;
- Compose deployment: project plus service name;
- nginx deployment: canonical config path;
- domain: normalized lowercase hostname without trailing dot;
- systemd service: unit name.
Container IDs and PIDs are observations, not stable entity IDs. Never use them
as the sole upsert key. Every deployment includes `last_seen_at`; only a full
successful section can mark previously observed entities missing.
## Collector Requirements
### Processes
- Group related processes deterministically by executable, working directory,
and parent relationship.
- Parse TCP and UDP listeners for IPv4 and IPv6.
- Do not read another process's environment.
- Exclude kernel threads and the worker itself from application suggestions.
### Docker And Compose
- Disabled without an explicit Docker capability.
- Prefer Docker/Compose JSON output and labels; do not parse human tables.
- Collect project, service, image, state, ports, mount paths, health, and
Traefik host rules. Exclude environment values and registry credentials.
- Treat Docker socket access as root-equivalent and report the capability to
the operator.
### Nginx
- Read only configured allowlisted roots.
- Collect config path, listen values, server names, auth presence, root path,
and proxy presence. Do not send certificate private-key paths or contents.
### Host And Server IPs
- Collect hostname, OS/kernel summary, and valid non-loopback addresses.
- Do not guess a primary address; report interface and route metadata so the
control plane can apply policy.
## Relationship To deploymentd And rstuff
Worker reports and deploymentd ingest are complementary producers of the same
control-plane projection. During migration:
- deploymentd remains authoritative for sections it currently reports;
- worker inventory is feature-gated per server;
- every row records source and source report ID;
- reconciliation occurs per source and section to prevent one producer from
deleting another producer's observations;
- the rstuff `rsmon.inventory.v1` stream remains a control-plane output/input,
not a worker protocol.
## Delivery Phases
1. Start and stop the existing process collector; move it behind an interface.
2. Add typed report and control-plane ingestion for process/host/address data.
3. Add read-only Docker and Compose discovery.
4. Add nginx and systemd discovery.
5. Add source-aware reconciliation and deploymentd comparison mode.
6. Enable monitor suggestions only after operators confirm discovered targets.
## Acceptance Tests
- A worker restart preserves no false stable identity based on PID/container ID.
- A failed Docker section does not erase successful nginx or process data.
- Partial reports never mark absent deployments missing.
- Full reports reconcile only rows owned by the same source and section.
- Cross-account and worker/server mismatch reports are rejected.
- Fixture snapshots prove no environment values or credential-like fields are
serialized.
- Worker and deploymentd fixtures normalize equivalent Compose/nginx entities
to the same control-plane identity without deleting each other's rows.

127
docs/network-diagnostics.md Обычный файл
Просмотреть файл

@@ -0,0 +1,127 @@
# Network Diagnostics
## Current Capability
Two different features must not be conflated:
1. **Cross-worker confirmation** is partially implemented. The control plane
can assign the same failed check to another operated worker, and peer
selfcheck/quorum helpers exist in `internal/distworker/peer.go`,
`consensus.go`, and `selfcheck.go`.
2. **Rich diagnostic tasks** (`diag_http`, `diag_ssh`, DNS, TCP, traceroute,
MTR) are not implemented as worker protocol kinds.
Confirmation remains a normal centrally leased check and does not require a
new executor. Rich diagnostics require typed tasks and structured timing/path
results.
## Security Boundary
Diagnostics can become a network scanner. Every diagnostic task must be bound
to a monitor target already authorized and normalized by the control plane.
The worker receives the normalized target plus a signed target hash and refuses
runtime overrides.
Required restrictions:
- private worker account must equal task account;
- source worker must not equal a target worker represented by the monitor;
- no arbitrary host, URL, resolver, port range, or shell arguments;
- resolve every hostname locally and reject prohibited IPs before connecting;
- revalidate every HTTP redirect target;
- enforce per-task timeout, output limit, concurrency, and destination rate;
- never return authorization headers, request bodies, response bodies, SSH
credentials, or environment data;
- audit allow and deny decisions on the control plane.
## Target Package
```text
internal/netdiag/
types.go
validate.go
http.go
ssh.go
tcp.go
dns.go
traceroute_linux.go
mtr.go
```
`internal/netdiag` accepts a validated immutable request and returns a bounded
result. It does not select workers, query monitors, persist data, or send
protocol frames. `internal/distworker` owns task dispatch and result transport.
## Protocol
Extend `wire.TaskEnvelope` with `type=diagnostic` and a dedicated diagnostic
branch. Do not encode diagnostics as check settings or untyped events.
Common request fields:
- job ID and exact lease token;
- account, monitor, source worker, optional target worker, and region IDs;
- kind and scheduling reason (`periodic`, `on_demand`, `failure_triggered`);
- normalized target and target hash;
- timeout and kind-specific bounded options.
Common result fields:
- job ID, lease token, kind, source and target IDs;
- start/finish time and total latency;
- `ok` or `err` status;
- normalized error class;
- bounded kind-specific data.
Error classes are `timeout`, `dns_error`, `tcp_refused`, `tcp_reset`,
`tls_handshake`, `tls_cert`, `http_status`, `auth_required`, `auth_failed`,
`protocol_error`, `target_denied`, `unsupported_kind`, and `internal_error`.
## Delivery Phases
### D1: HTTP Diagnostics
Implement GET/HEAD only. Capture DNS, connect, TLS, first-byte, total timing,
status, final URL, bounded redirect chain, selected response headers, response
size, TLS version/cipher, and certificate fingerprints. Do not return body
content. Clamp total timeout to 30 seconds.
### D2: SSH Diagnostics
Perform banner, key exchange, and host-key inspection without authentication.
Return DNS/connect/KEX timing, banner, selected algorithms, and SHA-256 host-key
fingerprint. Password and private-key authentication are not part of v1.
### D3: TCP And DNS
TCP performs one destination connect and distinguishes timeout, refused, and
reset. DNS resolves only the monitor hostname with the system or explicitly
allowlisted resolver and returns bounded A/AAAA/CNAME answers and timing.
### D4: Traceroute And MTR
Linux-only, capability-gated, and disabled for private workers until abuse
review. Use a native bounded implementation or a fixed executable with fixed
arguments; never pass user strings to a shell. Return a maximum hop count and
bounded probes per hop.
## Relationship To Raft
Normal diagnostic tasks remain centrally leased. Do not replicate raw
diagnostic results in Raft. If duplicate failure-triggered diagnostics become a
measured problem, the critical cluster may replicate only compact dedup
metadata; this is not a prerequisite for diagnostic v1.
## Acceptance Tests
- Existing confirmation excludes the original failing worker and handles
duplicate results idempotently.
- Diagnostic tasks cannot target a host or port different from their signed
monitor target.
- HTTP redirects to prohibited addresses are rejected.
- Private workers reject cross-account tasks before opening a socket.
- HTTP and SSH fixture servers produce deterministic phase timings and bounded
metadata.
- Timeouts cancel DNS/connect/TLS/read work and leave no goroutine behind.
- Unsupported kinds return a structured terminal result.
- Result serialization contains no credentials or response body.

147
docs/private-workers.md Обычный файл
Просмотреть файл

@@ -0,0 +1,147 @@
# Private Workers
## Model
Operated and customer-operated workers run the same binary and protocol. The
control plane assigns trust and scope. A private worker is untrusted for every
account except its own and cannot widen its own scope through capabilities or
protocol fields.
Current shared scheduler code already supports restrictive account selection:
- `app/models/worker_node.go` derives allowed check and notification accounts;
- `app/models/check_jobs.go` filters check leasing;
- `app/models/task_selector.go` filters generic task leasing;
- a nonempty account list is restrictive; an empty list means platform worker
and is never available to a customer worker.
This is only one isolation layer. The executable still trusts the authenticated
control plane's init/task payload and private-worker onboarding is incomplete.
## Required Invariants
- A private worker has one immutable `account_id` in signed runtime config.
- Every normal task includes `account_id`; the worker rejects mismatch before
execution and reports a protocol error without touching the target.
- Credentials are filtered by account and method on the control plane, then
checked again by exact credential ID in the worker.
- Private workers do not receive another account's monitor, contact, LLM, peer,
inventory, or notification data.
- A private worker cannot become operated by changing local config or reported
capabilities.
- Results cannot choose their account. The control plane applies them against
the leased task and authenticated worker.
- Local UI data stays on the worker unless a typed, bounded protocol explicitly
permits upload.
## Onboarding
The current production path uses a worker token created by an administrator.
The customer self-service target is:
1. Control plane creates a disabled private-worker record and a one-time,
15-minute bootstrap token.
2. Installer writes control-plane URL and bootstrap token to a mode-0600
temporary environment file.
3. Worker exchanges it over TLS for a long-lived worker token and signed
immutable account identity.
4. Worker writes the long-lived token to the configured secret file with mode
0600, removes the bootstrap token, and reconnects.
5. Control plane activates the worker only after the first authenticated
heartbeat and valid capability report.
The worker must never receive the control-plane registration admin secret.
Bootstrap replay, expiry, worker-ID mismatch, or account mismatch fails closed.
## Token Rotation
Rotation is a two-token handoff:
1. Control plane issues a replacement token and keeps the old token valid for a
short bounded overlap.
2. Worker atomically writes the replacement secret.
3. Worker closes only the active control-plane connection and reconnects.
4. Successful authentication acknowledges rotation; control plane revokes the
old token.
5. Failure retains the old token until overlap expires and emits an operator
warning.
The current `Runner.RotateToken` terminates the runner by closing its stop
channel. This must be fixed before claiming unattended rotation.
## Runtime Config Authentication
Private-worker init/config payloads add:
- worker ID and immutable account ID;
- monotonic config version;
- issue and expiry timestamps;
- allowed check kinds, notification methods, and public-task policy;
- credential-set hash;
- Ed25519 signature and key ID.
The worker pins the control-plane verification key at bootstrap. It rejects
signature failure, downgrade, expiry, account change, and unknown critical
fields. Credentials remain in memory and are cleared when their signed scope
expires.
## Public Checks
Cross-account public-check execution is not the same as normal private scope.
It requires a separately signed `public_tasks` grant with:
- explicit opt-in by worker owner;
- allowed methods limited to safe public probes, initially HTTP GET/HEAD;
- no credentials, custom authorization headers, request bodies, private IP
targets, or notification tasks;
- independent concurrency and rate limits;
- SSRF validation after every DNS resolution and redirect;
- auditable reward/usage identity owned by the control plane.
Do not represent public permission by making the private worker's account list
empty or adding arbitrary account IDs.
## Deployment Modes
The systemd and Compose packages are both supported. Capabilities, not install
type, control host access:
- base: normal network checks and delegated notifications;
- host metrics: procfs and statfs access;
- inventory: read-only process/system discovery;
- Docker discovery: explicit socket access, treated as root-equivalent;
- Compose mutation: separate high-risk capability, disabled by default;
- Raft voter: durable fsync-capable cluster data directory and mTLS transport.
`CAP_NET_RAW` is granted only for ping/traceroute features. The worker remains
unprivileged otherwise.
## Worker Self-Monitoring
A worker must not run a monitor that represents its own process or host. The
control plane excludes it during selection; the worker also rejects a task that
names its own worker/host identity once those fields are signed into the task.
Peer health checks and control-plane selfcheck are separate from customer
monitor execution.
## Implementation Work Packages
1. Add signed account/config identity to `internal/wire` and runner state.
2. Validate task account and credential scope locally before dispatch.
3. Implement one-time bootstrap and atomic token storage/rotation.
4. Add worker disable/revoke behavior and visible stale-config state.
5. Add mTLS as an optional first transport, then require it for Raft clusters.
6. Add public-task grant and SSRF-safe executor only after private isolation is
proven.
## Acceptance Tests
- A private worker never leases or executes another account's normal task.
- A forged capability or account field cannot widen scope.
- Invalid, expired, downgraded, or differently scoped signed config is rejected.
- Bootstrap tokens are single-use and absent from disk after exchange.
- Rotation reconnects without stopping web, inventory, metrics, or cluster.
- Credential snapshots contain only the worker account and allowed methods.
- Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix
sockets, or a private redirect target.
- Revocation prevents reconnect and clears in-memory credentials.

125
docs/source-plan-migration.md Обычный файл
Просмотреть файл

@@ -0,0 +1,125 @@
# Source Plan Migration Ledger
## Purpose
This ledger records how worker-related documents from the RSMon control-plane
repository were converted. The source plans are historical input, not runtime
authority. Their worker-owned requirements now live in this repository's
implementation documents; control-plane-owned requirements stay in RSMon.
Conversion date: 2026-07-13.
## Direct Worker Documents
| Former RSMon document | Worker implementation document | Resolution |
| --- | --- | --- |
| `docs/new-worker.md` | root `README.md`, `architecture.md`, `private-workers.md` | Replaced monorepo commands with standalone binary/container/systemd operation |
| `docs/worker-container.md` | root `README.md`, `implementation-roadmap.md` | Distribution ownership moved here; production image tags retained |
| `docs/worker-http-settings.md` | `web-console-and-observability.md`, root `README.md` | Current port/auth behavior retained; Compose-only `WORKER_BIND_IP` clarified |
| `docs/worker-protocol.md` | `control-plane-protocol.md` | Current tagged task envelope is normative; stale jobs/results envelopes rejected |
| `docs/worker-cluster-quickstart.md` | `critical-check-cluster.md` | Existing cluster scaffold separated from unimplemented critical engine |
| `docs/rsmon-distributed-api-plan.md` | `architecture.md`, `control-plane-protocol.md`, `implementation-roadmap.md` | PostgreSQL normal path retained; Redis/Rabbit/NATS queue proposal superseded |
| `docs/plans/worker-notifier-mvp.md` | `tasks-and-notifications.md` | Final `task_envelope` and credential implementation supersede older sibling-field drafts |
| `docs/sessions/monitor-transfer-and-admin-workers.md` | `architecture.md`, `private-workers.md` | Historical operator/session decisions reduced to current ownership and account-isolation rules |
## Distributed Plans
| Former RSMon document | Worker implementation document | Resolution |
| --- | --- | --- |
| `docs/distributed/notifications-from-worker.md` | `tasks-and-notifications.md`, `private-workers.md` | Implemented methods/statuses recorded; old credential assumptions replaced |
| `docs/distributed/private-workers.md` | `private-workers.md` | Scheduler isolation marked current; bootstrap, signed config, and mTLS remain explicit work |
| `docs/distributed/network-diagnostics.md` | `network-diagnostics.md` | Current confirmation separated from future rich diagnostic tasks |
| `docs/distributed/worker-to-worker-raft.md` | `critical-check-cluster.md` | Converted to package-specific ordered implementation and release gates |
| `docs/distributed/critical-check-amendments.md` | `critical-check-cluster.md` | No-leases, four-quorum, external-witness, and commit-before-notify rules made normative |
| `docs/distributed/worker-web-app.md` | `web-console-and-observability.md`, `inventory.md` | Existing console recognized; future OAuth/Compose/hardware scope split into staged work |
## Inventory And Host Documents
| Former RSMon document | Worker implementation document | Resolution |
| --- | --- | --- |
| `docs/inventory-management.md` | `inventory.md` | Control-plane projection retained as dependency; worker owns collection only |
| `docs/plans/inventory-management.md` | `inventory.md` | deploymentd and worker reports defined as source-aware complementary producers |
| `docs/parity/rstuff-inventory.md` | `inventory.md` | rstuff enums/stream remain control-plane contracts, not worker protocol |
| `docs/parity/deploymentd.md` | `inventory.md` | deploymentd parity becomes fixture normalization and comparison gate |
| `docs/deploymentd-replay-rollout.md` | `inventory.md` | HTTP nonce rollout remains deploymentd-specific; worker uses authenticated typed reports |
| `docs/integrations/rstuff-inventory-stream.md` | `inventory.md` | Valkey stream remains between control plane and rstuff, never worker transport |
| `docs/plans/servers-and-hardware-metrics.md` | `web-console-and-observability.md`, `inventory.md` | Implemented server metric report retained; local hardware extensions staged |
| `docs/server-observability.md` | `web-console-and-observability.md`, `control-plane-protocol.md` | Current wire fields, ownership validation, and TSDB boundary retained |
| `docs/parity/capture.md` | `web-console-and-observability.md`, `implementation-roadmap.md` | Worker-host coverage gaps converted into staged collector work |
## Diagnostics, Policy, And Product Documents
| Former RSMon document | Worker implementation document | Resolution |
| --- | --- | --- |
| `docs/network-diagnostics.md` | `network-diagnostics.md` | Implemented confirmation/quarantine baseline retained |
| `docs/plans/network-diagnostics-partial.md` | `network-diagnostics.md` | Partial implementation folded into explicit rich-task phases |
| `docs/notification-task-delivery.md` | `tasks-and-notifications.md`, `control-plane-protocol.md` | Durable queue remains control-plane-owned; attempt execution remains worker-owned |
| `docs/notification-credentials.md` | `tasks-and-notifications.md`, `private-workers.md` | SMTP/Telegram current scope retained; webhook/Mattermost gaps explicit |
| `docs/billing-and-entitlements.md` | `private-workers.md`, `implementation-roadmap.md` | Entitlement remains a control-plane gate, never trusted from worker config |
| `docs/plans/public-checks.md` | `private-workers.md`, `implementation-roadmap.md` | Public tasks use a separate safe grant, not widened account access |
| `docs/plans/plans-and-billing.md` | `private-workers.md` | Plan eligibility remains control-plane policy |
| `docs/check-kinds.md` | `tasks-and-notifications.md` | Worker-supported executor list tied to `internal/checkexec` |
| `docs/plans/uptimerobot-parity.md` | `implementation-roadmap.md` | Worker-owned parity dependencies ordered by security prerequisites |
| `docs/plans/krasichka.md` | none | External Krasichka runtime is not a feature of this worker binary; only protocol compatibility applies |
## Operations And CI Documents
| Former RSMon document | New authority | Resolution |
| --- | --- | --- |
| `docs/local-processes.md` | root `README.md` | Standalone development commands replace monorepo process instructions |
| `docs/ci.md` | `.github/workflows/*.yml`, `implementation-roadmap.md` | Worker image build is owned by Gitea Actions here |
| `docs/CI_QUICKSTART.md` | root `README.md` | Standalone make, Docker, and Compose commands are authoritative |
## Resolved Conflicts
### Protocol
Old plans used `type: jobs/results`, separate notification siblings, and
version-string feature detection. Current code uses `kind`, explicit branches,
`task_envelope`, exact lease tokens, and explicit capabilities. The current
model is authoritative.
### Queue Ownership
Normal task ownership is PostgreSQL with `FOR UPDATE SKIP LOCKED`. Redis
Streams, RabbitMQ, and NATS are not worker dependencies. The critical path is
dispatchless Raft and cannot reuse normal leases.
### Credentials
SMTP and Telegram use scoped pushed credential lists. Webhook and Mattermost
are not yet equivalently persisted per account. SMS and voice are unsupported.
No document may claim universal per-account provider isolation before those
gaps close.
### Web Console
The console exists on port 27401 with Basic auth or local bcrypt. The former
draft's port 7401, OAuth-first behavior, and entirely unimplemented status are
obsolete. Collector lifecycle remains a real implementation gap.
### Inventory
deploymentd HTTP ingest is currently implemented on the control plane. Worker
inventory is local/partial and has no upstream report yet. Future worker and
deploymentd reports are complementary source-scoped inputs; neither may erase
the other's observations.
### Private Workers And Public Work
Private workers are account-restricted. Public cross-account checks require a
separate signed and SSRF-safe grant. Empty account scope continues to mean a
platform-operated worker and must never be customer-selectable.
### Raft Maturity
Membership, persistence, snapshots, transport, and placeholder FSM entries
exist. Production `distributed_critical` behavior does not. The control plane
is an external witness, never a voter; provider calls remain outside Raft.
## Maintenance
When a new RSMon plan adds worker behavior, update this ledger and one concrete
implementation document in the same change. Do not copy a planning draft into
this repository without reconciling it against current packages, wire types,
security boundaries, and tests.

131
docs/tasks-and-notifications.md Обычный файл
Просмотреть файл

@@ -0,0 +1,131 @@
# Normal Tasks And Notifications
## Normal Check Execution
Normal work is centrally scheduled. The control plane owns PostgreSQL task
rows, selection, leases, retries, deadlines, and dead-letter state. The worker
owns only one execution attempt and its terminal report.
The implemented path is:
1. `internal/distworker/client.go` opens the authenticated WebSocket.
2. `internal/distworker/runner.go` selects a recognized populated task branch.
3. A bounded worker pool executes the task under its deadline.
4. `internal/checkexec/exec.go` dispatches to a database-free check package.
5. The runner queues one result and the result writer returns it to the control
plane with the exact lease token.
Implemented check kinds are HTTP, SSL, SSH, FTP, DNS, WHOIS, BSSL, LLM,
LLM-HTTP, ping, TCP, and UDP. Control-plane selection uses capabilities. The
local dispatcher supports only its explicit switch cases, but strict malformed
envelope and unsupported-kind result handling remains work package N1 rather
than a complete fail-closed protocol response.
## Queue And Shutdown Rules
- Input and output queues are bounded. Backpressure must not create unbounded
goroutines or memory use.
- Concurrency is supplied by the control plane but clamped by the local
maximum.
- A task panic is recovered at the task boundary and reported as a failed
attempt; it must not kill the runner.
- SIGTERM stops accepting tasks, allows bounded in-flight completion, attempts
final result delivery, and then exits.
- WebSocket reconnect does not re-run an in-flight or completed task.
- HTTP polling remains compatibility-only and must not become a second normal
scheduler.
## Delegated Notifications
`internal/distworker/notification.go` executes one pre-rendered delivery
attempt. It does not render alert policy and does not own retry scheduling.
| Method | State | Credential source |
| --- | --- | --- |
| Email | Implemented | scoped SMTP list from init/config |
| Telegram | Implemented | scoped bot list from init/config |
| Webhook | Implemented | current shared signing configuration |
| Mattermost | Implemented | current shared defaults/task endpoint |
| SMS | Explicitly unsupported | no provider selected |
| Voice | Explicitly unsupported | no provider selected |
SMS and voice return `permanent` with `unsupported_method`; they must never be
reported as delivered or silently dropped.
Worker result statuses are `delivered`, `retryable`, `permanent`, and
`partial`. Transport failures, provider rate limits, and provider 5xx responses
are retryable. Invalid recipients, authentication failures, malformed payloads,
and unsupported methods are permanent. Positive provider retry-after values
are included in the result; the control plane owns the actual retry time.
## Credentials
Credentials arrive in `WorkerInit.Credentials` and remain in memory. SMTP and
Telegram tasks may request a credential ID; the worker resolves only that ID
from the authorized pushed set. Missing IDs are permanent failures. Do not
fall back to another account's or platform credential.
The following fields must be redacted from logs and UI buffers: password,
token, API key, signing secret, hook URL, authorization headers, and fields
ending in `_secret`. Provider response text is bounded and must not include
response bodies that could contain secrets.
Webhook and Mattermost need persisted per-account credentials in the control
plane before they can claim the same isolation guarantees as SMTP and
Telegram. The worker wire shape already separates method configuration; do not
introduce environment-global fallback for private workers.
## Selfcheck Notifications
System selfcheck alerts are not normal notification tasks. They allow a worker
to notify configured system contacts when the control plane itself is
unreachable. They use system contacts and credentials from config, local
jitter, and local deduplication. They must not impersonate customer delivery or
mutate control-plane message state.
The local notifications page currently records selfcheck deliveries. Normal
delegated delivery attempts should be added to the same bounded view after
redaction, with job ID, method, status, duration, and time only.
## Implementation Work Packages
### N1: Runner Integration Coverage
Add strict envelope/inner job-ID validation and structured unsupported-kind
results. Add WebSocket integration tests for reconnect, task panic recovery,
queue backpressure, result resend, stale lease behavior, malformed envelopes,
unsupported kinds, and graceful drain. Target
`internal/distworker/runner_protocol_test.go` and a local test WebSocket server.
### N2: Complete Local Delivery Audit
Record delegated notification outcomes in the bounded local notification ring.
Never store recipient values, rendered body, credentials, or full provider
responses in SQLite. Verify `/notifications` shows both selfcheck and delegated
attempts without leaking secrets.
### N3: Per-Account Webhook And Mattermost Credentials
Once the control plane sends credential IDs and account-scoped records, require
an exact credential match just like SMTP and Telegram. Remove global fallback
for private workers. Add cross-account negative tests.
### N4: SMS And Voice Providers
Select a provider and define a credential wire type before adding an executor.
Implementation must include timeout, retry classification, provider response
redaction, idempotency support, and contract tests. Until all are present,
retain the explicit permanent unsupported result.
## Acceptance Tests
- Every supported check kind can execute from a serialized task without a
database connection.
- A stale or missing lease token cannot produce an accepted normal result.
- One task causes at most one local execution during reconnect and duplicate
frame delivery.
- Private workers cannot resolve a credential outside their authorized set.
- SMTP 4xx/rate-limit conditions are retryable; invalid authentication is
permanent.
- SMS and voice remain explicit failures until provider tests exist.
- Logs and local UI contain no pushed credential values.

128
docs/web-console-and-observability.md Обычный файл
Просмотреть файл

@@ -0,0 +1,128 @@
# Web Console And Host Observability
## Current Implementation
`internal/webapp` is a functional embedded HTTP application backed by SQLite.
It provides authenticated overview, apps, checks, notifications, logs, status,
settings, updates, password rotation, token rotation, audit retention, and
cluster status endpoints. Static assets and templates are embedded in the
worker binary.
The listener defaults to `WORKER_HOST=0.0.0.0` and `WORKER_PORT=27401`.
Compose publishes it on host loopback by default through `WORKER_BIND_IP`; that
variable is a Compose interpolation setting, not a variable read by the binary.
## Authentication
Two modes are implemented:
| Configuration | Behavior |
| --- | --- |
| Both `WORKER_LOGIN` and `WORKER_PASSWORD` set | Basic-auth-backed operator sessions and `/web/api/*` access |
| Both empty | Local bcrypt user; a one-time password is printed on first start |
| Only one set | Startup error |
Sessions are HTTP-only and SameSite strict, with a 30-minute idle and 8-hour
absolute lifetime. State-changing browser requests require CSRF validation.
Authenticated responses use no-store and restrictive security headers.
OAuth/device authorization is not implemented. Public exposure requires a TLS
reverse proxy and network restrictions. Do not claim that `WORKER_URL` enables
TLS; it only advertises the externally reachable URL.
## Known Runtime Gap
`webapp.New` constructs `Inventory` and `Metrics`, but `Server.Start` currently
does not start either collector and `Server.Close` does not stop them. As a
result, production `/apps` and `/status` pages can remain empty even though
collector unit tests pass.
This is the first required web-console change:
1. Start both collectors with the server context before accepting requests.
2. Stop and wait for them during `Close` and context cancellation.
3. Make collector start idempotent and collector failure observable.
4. Add a server integration test that observes populated inventory and metrics
after startup.
## Inventory View
The current local collector reads Linux `/proc`, process command lines and
working directories, TCP listeners, and resource data, then stores snapshots
in SQLite. It does not yet provide complete grouping, Docker, Compose, systemd,
nginx, UDP, or deploymentd-compatible inventory. The implementation plan is in
[inventory.md](inventory.md).
## Host Metrics
There are two related collectors:
- `internal/distworker/server_metrics.go` sends the linked server's CPU,
memory, disk, load, uptime, process, and network snapshot to the control
plane every five seconds. This path is implemented.
- `internal/webapp/metrics.go` drives local status pages and a local ring. It
exists but is affected by the lifecycle gap above.
The control-plane report uses interval rates for CPU and aggregate networking.
The local metrics implementation currently computes CPU from lifetime totals;
align it with interval deltas so local and remote views agree.
## Compose Management Target
Compose management remains planned and is opt-in. Implement it after read-only
inventory is complete, under a separate `internal/composeops` package.
Required constraints:
- fixed stacks root, default `/opt/stacks`, with canonical path containment;
- explicit `docker compose -f <validated-file>` invocation;
- no shell interpolation and no free-form host command endpoint;
- per-stack single-flight action lock;
- async `202 Accepted` operations with bounded progress streaming;
- viewer/operator/admin permissions and audit records;
- typed confirmation for down, delete, and restore;
- compose validation before save and backup before deploy;
- Docker socket treated as root-equivalent and disabled by default.
The first Compose release includes read/list/status/logs only. Mutation, file
editing, environment editing, terminal access, backup, and restore are separate
release gates rather than one large feature switch.
## Host Status Extensions
Implement in this order:
1. Correct `/proc`, `statfs`, and `/proc/net/dev` interval collection.
2. Add bounded local 24-hour one-minute aggregates.
3. Add optional `lsblk --json` device topology.
4. Add optional cached SMART and sensors data.
5. Add Docker daemon and per-container status only when the socket capability
is enabled.
Missing commands or permissions produce unavailable sections, not worker
startup failure. All subprocesses use fixed argument arrays, deadlines, output
limits, and allowlisted executable names.
## Health And Updates
`GET /healthz` is local process liveness. The image healthcheck should call
this endpoint, not `rsmon-worker health`, because the latter probes the control
plane `/up` endpoint. Control-plane outage is a selfcheck/readiness event, not
proof that the worker process is dead.
In-place binary replacement from the web UI is not implemented and should not
run with ambient root access. The systemd update flow should download a signed
artifact to a staging path and require an external privileged installer or
explicit operator command to activate it.
## Acceptance Tests
- Starting `webapp.Server` produces nonempty metrics and process inventory on
Linux and stops every collector cleanly.
- Both auth modes work, XOR credentials fail startup, CSRF protects writes,
and unauthenticated API calls fail.
- A public bind is documented as insecure without TLS; no credential is logged.
- `/healthz` remains healthy during a simulated control-plane outage.
- Every host command has timeout/output-limit tests and rejects user-supplied
executable or path traversal.
- Compose mutation cannot operate outside the configured stacks root.