worker/docs/private-workers.md

Private Workers

Model

Operated and customer-operated workers run the same binary and protocol. The control plane assigns trust and scope. A private worker is untrusted for every account except its own and cannot widen its own scope through capabilities or protocol fields.

Current shared scheduler code already supports restrictive account selection:

• app/models/worker_node.go derives allowed check and notification accounts;
• app/models/check_jobs.go filters check leasing;
• app/models/task_selector.go filters generic task leasing;
• a nonempty account list is restrictive; an empty list means platform worker and is never available to a customer worker.

This is only one isolation layer. The executable still trusts the authenticated control plane's init/task payload and private-worker onboarding is incomplete.

Required Invariants

• A private worker has one immutable account_id in signed runtime config.
• Every normal task includes account_id; the worker rejects mismatch before execution and reports a protocol error without touching the target.
• Credentials are filtered by account and method on the control plane, then checked again by exact credential ID in the worker.
• Private workers do not receive another account's monitor, contact, LLM, peer, inventory, or notification data.
• A private worker cannot become operated by changing local config or reported capabilities.
• Results cannot choose their account. The control plane applies them against the leased task and authenticated worker.
• Local UI data stays on the worker unless a typed, bounded protocol explicitly permits upload.

Onboarding

The current production path uses a worker token created by an administrator. The customer self-service target is:

1. Control plane creates a disabled private-worker record and a one-time, 15-minute bootstrap token.
2. Installer writes control-plane URL and bootstrap token to a mode-0600 temporary environment file.
3. Worker exchanges it over TLS for a long-lived worker token and signed immutable account identity.
4. Worker writes the long-lived token to the configured secret file with mode 0600, removes the bootstrap token, and reconnects.
5. Control plane activates the worker only after the first authenticated heartbeat and valid capability report.

The worker must never receive the control-plane registration admin secret. Bootstrap replay, expiry, worker-ID mismatch, or account mismatch fails closed.

Token Rotation

Rotation is a two-token handoff:

1. Control plane issues a replacement token and keeps the old token valid for a short bounded overlap.
2. Worker atomically writes the replacement secret.
3. Worker closes only the active control-plane connection and reconnects.
4. Successful authentication acknowledges rotation; control plane revokes the old token.
5. Failure retains the old token until overlap expires and emits an operator warning.

The current Runner.RotateToken terminates the runner by closing its stop channel. This must be fixed before claiming unattended rotation.

Runtime Config Authentication

Private-worker init/config payloads add:

• worker ID and immutable account ID;
• monotonic config version;
• issue and expiry timestamps;
• allowed check kinds, notification methods, and public-task policy;
• credential-set hash;
• Ed25519 signature and key ID.

The worker pins the control-plane verification key at bootstrap. It rejects signature failure, downgrade, expiry, account change, and unknown critical fields. Credentials remain in memory and are cleared when their signed scope expires.

Public Checks

Cross-account public-check execution is not the same as normal private scope. It requires a separately signed public_tasks grant with:

• explicit opt-in by worker owner;
• allowed methods limited to safe public probes, initially HTTP GET/HEAD;
• no credentials, custom authorization headers, request bodies, private IP targets, or notification tasks;
• independent concurrency and rate limits;
• SSRF validation after every DNS resolution and redirect;
• auditable reward/usage identity owned by the control plane.

Do not represent public permission by making the private worker's account list empty or adding arbitrary account IDs.

Deployment Modes

The systemd and Compose packages are both supported. Capabilities, not install type, control host access:

• base: normal network checks and delegated notifications;
• host metrics: procfs and statfs access;
• inventory: read-only process/system discovery;
• Docker discovery: explicit socket access, treated as root-equivalent;
• Compose mutation: separate high-risk capability, disabled by default;
• Raft voter: durable fsync-capable cluster data directory and mTLS transport.

The simple systemd installer currently runs the worker as root, matching the minimal host-install model. Docker runs with the image's unprivileged user and adds NET_RAW for ping/traceroute. A future hardened systemd profile can use a dedicated user and narrow capabilities when host inventory requirements are finalized.

The standalone binary provides two systemd installation paths:

• rsmon-worker install installs the current binary locally and writes the worker URL/token to a root-owned mode-0600 environment file; --docker installs a systemd-managed prebuilt image instead;
• rsmon-worker deploy verifies an SSH host key, uploads the binary and a temporary mode-0600 environment file, and invokes install remotely; --docker uploads only the environment and unit, then pulls on the target.

Both default to https://rsmon.ru and accept --token-file for automation. Direct secret flags are supported but can be visible in process listings; file options are preferred. SSH login credentials, sudo credentials, and the worker token remain separate.

Worker Self-Monitoring

A worker must not run a monitor that represents its own process or host. The control plane excludes it during selection; the worker also rejects a task that names its own worker/host identity once those fields are signed into the task. Peer health checks and control-plane selfcheck are separate from customer monitor execution.

Implementation Work Packages

1. Add signed account/config identity to internal/wire and runner state.
2. Validate task account and credential scope locally before dispatch.
3. Implement one-time bootstrap and atomic token storage/rotation.
4. Add worker disable/revoke behavior and visible stale-config state.
5. Add mTLS as an optional first transport, then require it for Raft clusters.
6. Add public-task grant and SSRF-safe executor only after private isolation is proven.

Acceptance Tests

• A private worker never leases or executes another account's normal task.
• A forged capability or account field cannot widen scope.
• Invalid, expired, downgraded, or differently scoped signed config is rejected.
• Bootstrap tokens are single-use and absent from disk after exchange.
• Rotation reconnects without stopping web, inventory, metrics, or cluster.
• Credential snapshots contain only the worker account and allowed methods.
• Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix sockets, or a private redirect target.
• Revocation prevents reconnect and clears in-memory credentials.