cb23f123ae
Все проверки выполнены успешно
CI / test (push) Successful in 10m15s
Docker / Build and publish worker image (push) Successful in 34m59s
92 строки
5.1 KiB
Markdown
92 строки
5.1 KiB
Markdown
# Worker Architecture
|
|
|
|
## Runtime
|
|
|
|
`cmd/rsmon-worker/main.go` composes three independently stoppable subsystems:
|
|
|
|
1. `internal/distworker`: persistent control-plane connection, normal task
|
|
execution, selfcheck, notification delivery, and server metric reports.
|
|
2. `internal/webapp`: authenticated local operator console and SQLite-backed
|
|
local state.
|
|
3. `internal/workercluster`: optional worker-to-worker Raft transport and FSM.
|
|
|
|
All three share one cancellation context. The worker can run without the web
|
|
console using `--no-web`; the cluster is disabled unless
|
|
`WORKER_CLUSTER_ENABLED=true`.
|
|
|
|
## Two Execution Paths
|
|
|
|
The two check paths are deliberately separate.
|
|
|
|
| Path | Work origin | Ownership | Worker implementation |
|
|
| --- | --- | --- | --- |
|
|
| Normal checks and notifications | Control-plane PostgreSQL `tasks` rows | Exact lease token and expiry held by control plane | `internal/distworker` |
|
|
| `distributed_critical` | Signed config adopted by a worker Raft cluster | Raft FSM incident and outbox metadata | `internal/workercluster` plus a planned scheduler bridge |
|
|
|
|
Normal work always uses control-plane dispatch and leases. The critical path
|
|
must never use a normal task row, job assignment, or lease. Adding the critical
|
|
path must not change normal protocol behavior.
|
|
|
|
## Package Boundaries
|
|
|
|
| Package | Responsibility | Must not do |
|
|
| --- | --- | --- |
|
|
| `internal/wire` | JSON-compatible protocol types | Network or persistence work |
|
|
| `internal/distworker` | Connections, queues, pools, execution orchestration | Direct control-plane database writes |
|
|
| `internal/checkexec` | Database-free check dispatch | Scheduling or result persistence |
|
|
| `checks/*` | Protocol-specific probes | Worker selection or account policy |
|
|
| `internal/webapp` | Local UI, sessions, audit, local snapshots | Become the fleet source of truth |
|
|
| `internal/workercluster` | Raft transport, log, FSM, snapshots, membership | Store raw samples or plaintext credentials |
|
|
| `internal/sender` | Provider calls shared by worker executors | Own retries or durable task state |
|
|
|
|
New host discovery should move into `internal/inventory`; the web console may
|
|
read it through an interface but must not remain the collector's owner. New
|
|
network probes should live in `internal/netdiag` and use an allowlisted target
|
|
provided by a validated control-plane task.
|
|
|
|
## Current Status
|
|
|
|
| Capability | State | Evidence |
|
|
| --- | --- | --- |
|
|
| WebSocket normal check execution | Implemented | `internal/distworker/client.go`, `runner.go` |
|
|
| Check kinds | Implemented | `internal/checkexec/exec.go`, `checks/*` |
|
|
| Delegated notifications | Partial | email, Telegram, webhook, Mattermost implemented; SMS/voice unsupported |
|
|
| Host telemetry to control plane | Implemented | `internal/distworker/server_metrics.go` |
|
|
| Local web console | Implemented | `internal/webapp/server.go`, `routes.go` |
|
|
| Local inventory and status pages | Partial | collectors exist but are not started by `Server.Start` |
|
|
| Private-worker scheduler isolation | Implemented in shared selector code | `app/models/worker_node.go`, `task_selector.go`, `check_jobs.go` |
|
|
| Private-worker bootstrap and worker-side scope verification | Planned | no one-time bootstrap or signed config path |
|
|
| Cross-worker confirmation | Partial | peer selfcheck and shared model logic exist; no rich diagnostic tasks |
|
|
| Raft membership and persistence | Partial | `internal/workercluster` |
|
|
| Dispatchless critical checks | Planned | no scheduler, signature adoption, quorum incident engine, or outbox executor |
|
|
|
|
## Data Rules
|
|
|
|
- Credentials delivered in `init`/`config` stay in memory and are never written
|
|
to the local SQLite store, Raft log, or snapshots.
|
|
- Raw check samples and high-volume telemetry never enter Raft.
|
|
- Local UI state is not authoritative control-plane state.
|
|
- A normal result is not valid without the current exact lease token.
|
|
- Any worker-originated account, server, monitor, or target identifier is
|
|
untrusted until the control plane validates it against the authenticated
|
|
worker.
|
|
- A worker must not execute a synthetic health check whose target is itself.
|
|
|
|
## Runtime Completion Criteria
|
|
|
|
- SIGTERM drains the runner, stops collectors, closes listeners, shuts down
|
|
Raft, and closes SQLite without exceeding the service stop timeout.
|
|
- A web-console failure is reported and causes an intentional process policy;
|
|
it must not silently leave a partially healthy process.
|
|
- Startup validates the advertised origin: canonical `PUBLIC_URL` is held to
|
|
the strict scheme-and-authority shape (no userinfo, query, fragment, or
|
|
ambiguous path) and plain HTTP on a non-loopback host is rejected in an
|
|
explicitly production environment; the legacy `WORKER_URL` is read as a
|
|
bounded-migration fallback, held only to the tolerant absolute-URL check,
|
|
and logged with a deprecation warning. Both reject a missing hostname, e.g.
|
|
`https://:27401`. Startup also rejects incomplete auth credentials, invalid
|
|
cluster settings, and unwritable data directories before accepting work.
|
|
- `GET /healthz` reports process-local liveness. Control-plane reachability is
|
|
a separate readiness/selfcheck signal and must not make a healthy container
|
|
fail its local liveness probe.
|