worker/docs/architecture.md

Worker Architecture

Runtime

cmd/rsmon-worker/main.go composes three independently stoppable subsystems:

1. internal/distworker: persistent control-plane connection, normal task execution, selfcheck, notification delivery, and server metric reports.
2. internal/webapp: authenticated local operator console and SQLite-backed local state.
3. internal/workercluster: optional worker-to-worker Raft transport and FSM.

All three share one cancellation context. The worker can run without the web console using --no-web; the cluster is disabled unless WORKER_CLUSTER_ENABLED=true.

Two Execution Paths

The two check paths are deliberately separate.

Path	Work origin	Ownership	Worker implementation
Normal checks and notifications	Control-plane PostgreSQL tasks rows	Exact lease token and expiry held by control plane	internal/distworker
distributed_critical	Signed config adopted by a worker Raft cluster	Raft FSM incident and outbox metadata	internal/workercluster plus a planned scheduler bridge

Normal work always uses control-plane dispatch and leases. The critical path must never use a normal task row, job assignment, or lease. Adding the critical path must not change normal protocol behavior.

Package Boundaries

Package	Responsibility	Must not do
internal/wire	JSON-compatible protocol types	Network or persistence work
internal/distworker	Connections, queues, pools, execution orchestration	Direct control-plane database writes
internal/checkexec	Database-free check dispatch	Scheduling or result persistence
checks/*	Protocol-specific probes	Worker selection or account policy
internal/webapp	Local UI, sessions, audit, local snapshots	Become the fleet source of truth
internal/workercluster	Raft transport, log, FSM, snapshots, membership	Store raw samples or plaintext credentials
internal/sender	Provider calls shared by worker executors	Own retries or durable task state

New host discovery should move into internal/inventory; the web console may read it through an interface but must not remain the collector's owner. New network probes should live in internal/netdiag and use an allowlisted target provided by a validated control-plane task.

Current Status

Capability	State	Evidence
WebSocket normal check execution	Implemented	internal/distworker/client.go, runner.go
Check kinds	Implemented	internal/checkexec/exec.go, checks/*
Delegated notifications	Partial	email, Telegram, webhook, Mattermost implemented; SMS/voice unsupported
Host telemetry to control plane	Implemented	internal/distworker/server_metrics.go
Local web console	Implemented	internal/webapp/server.go, routes.go
Local inventory and status pages	Partial	collectors exist but are not started by Server.Start
Private-worker scheduler isolation	Implemented in shared selector code	app/models/worker_node.go, task_selector.go, check_jobs.go
Private-worker bootstrap and worker-side scope verification	Planned	no one-time bootstrap or signed config path
Cross-worker confirmation	Partial	peer selfcheck and shared model logic exist; no rich diagnostic tasks
Raft membership and persistence	Partial	internal/workercluster
Dispatchless critical checks	Planned	no scheduler, signature adoption, quorum incident engine, or outbox executor

Data Rules

• Credentials delivered in init/config stay in memory and are never written to the local SQLite store, Raft log, or snapshots.
• Raw check samples and high-volume telemetry never enter Raft.
• Local UI state is not authoritative control-plane state.
• A normal result is not valid without the current exact lease token.
• Any worker-originated account, server, monitor, or target identifier is untrusted until the control plane validates it against the authenticated worker.
• A worker must not execute a synthetic health check whose target is itself.

Runtime Completion Criteria

• SIGTERM drains the runner, stops collectors, closes listeners, shuts down Raft, and closes SQLite without exceeding the service stop timeout.
• A web-console failure is reported and causes an intentional process policy; it must not silently leave a partially healthy process.
• Startup currently rejects malformed WORKER_URL; the accepted target is PUBLIC_URL with bounded compatibility migration. Startup also rejects incomplete auth credentials, invalid cluster settings, and unwritable data directories before accepting work.
• GET /healthz reports process-local liveness. Control-plane reachability is a separate readiness/selfcheck signal and must not make a healthy container fail its local liveness probe.