worker/docs/private-workers.md

# Private Workers

## Model

Operated and customer-operated workers run the same binary and protocol. The
control plane assigns trust and scope. A private worker is untrusted for every
account except its own and cannot widen its own scope through capabilities or
protocol fields.

Current shared scheduler code already supports restrictive account selection:

- `app/models/worker_node.go` derives allowed check and notification accounts;
- `app/models/check_jobs.go` filters check leasing;
- `app/models/task_selector.go` filters generic task leasing;
- a nonempty account list is restrictive; an empty list means platform worker
  and is never available to a customer worker.

This is only one isolation layer. The executable still trusts the authenticated
control plane's init/task payload and private-worker onboarding is incomplete.

## Required Invariants

- A private worker has one immutable `account_id` in signed runtime config.
- Every normal task includes `account_id`; the worker rejects mismatch before
  execution and reports a protocol error without touching the target.
- Credentials are filtered by account and method on the control plane, then
  checked again by exact credential ID in the worker.
- Private workers do not receive another account's monitor, contact, LLM, peer,
  inventory, or notification data.
- A private worker cannot become operated by changing local config or reported
  capabilities.
- Results cannot choose their account. The control plane applies them against
  the leased task and authenticated worker.
- Local UI data stays on the worker unless a typed, bounded protocol explicitly
  permits upload.

## Onboarding

The current production path uses a worker token created by an administrator.
The customer self-service target is:

1. Control plane creates a disabled private-worker record and a one-time,
   15-minute bootstrap token.
2. Installer writes control-plane URL and bootstrap token to a mode-0600
   temporary environment file.
3. Worker exchanges it over TLS for a long-lived worker token and signed
   immutable account identity.
4. Worker writes the long-lived token to the configured secret file with mode
   0600, removes the bootstrap token, and reconnects.
5. Control plane activates the worker only after the first authenticated
   heartbeat and valid capability report.

The worker must never receive the control-plane registration admin secret.
Bootstrap replay, expiry, worker-ID mismatch, or account mismatch fails closed.

## Token Rotation

The current worker receives a replacement token in memory. In-memory reconnect
rotation is implemented: the control plane
invalidates the old token immediately, then `Runner.RotateToken` closes only
the active control-plane connection and reconnects with the replacement token.
It does not stop the runner or its web, inventory, metrics, or cluster
subsystems. There is no durable token write, overlap, rollback, or rotation
acknowledgement contract yet; persistent replacement-token failures can require
operator action.

If a websocket write fails after the worker has dequeued a check or notification
result, it resends that envelope after reconnect. Delivery is therefore
at-least-once; the control plane must deduplicate by the leased job and lease
token before applying a resent result. Failed server-metric snapshots are
dropped because they have no idempotency key and the next periodic collection
replaces them. Metric snapshots are bound to the active control connection;
snapshots collected while disconnected or for an older connection are dropped.

## Runtime Config Authentication

Private-worker init/config payloads add:

- worker ID and immutable account ID;
- monotonic config version;
- issue and expiry timestamps;
- allowed check kinds, notification methods, and public-task policy;
- credential-set hash;
- Ed25519 signature and key ID.

The worker pins the control-plane verification key at bootstrap. It rejects
signature failure, downgrade, expiry, account change, and unknown critical
fields. Credentials remain in memory and are cleared when their signed scope
expires.

## Public Checks

Cross-account public-check execution is not the same as normal private scope.
It requires a separately signed `public_tasks` grant with:

- explicit opt-in by worker owner;
- allowed methods limited to safe public probes, initially HTTP GET/HEAD;
- no credentials, custom authorization headers, request bodies, private IP
  targets, or notification tasks;
- independent concurrency and rate limits;
- SSRF validation after every DNS resolution and redirect;
- auditable reward/usage identity owned by the control plane.

Do not represent public permission by making the private worker's account list
empty or adding arbitrary account IDs.

## Deployment Modes

The systemd and Compose packages are both supported. Capabilities, not install
type, control host access:

- base: normal network checks and delegated notifications;
- host metrics: procfs and statfs access;
- inventory: read-only process/system discovery;
- Docker discovery: explicit socket access, treated as root-equivalent;
- Compose mutation: separate high-risk capability, disabled by default;
- Raft voter: durable fsync-capable cluster data directory and mTLS transport.

The simple systemd installer currently runs the worker as root, matching the
minimal host-install model. Docker runs with the image's unprivileged user and
adds `NET_RAW` for ping/traceroute. A future hardened systemd profile can use a
dedicated user and narrow capabilities when host inventory requirements are
finalized.

The standalone binary provides two systemd installation paths:

- `rsmon-worker install` installs the current binary locally and writes the
  worker URL/token to a root-owned mode-0600 environment file; `--docker`
  installs a systemd-managed prebuilt image instead;
- `rsmon-worker deploy` verifies an SSH host key, uploads the binary and a
  temporary mode-0600 environment file, and invokes `install` remotely;
  `--docker` uploads only the environment and unit, then pulls an explicitly
  supplied immutable image digest on the target.

Both default to `https://rsmon.ru` and accept `--token-file` for automation.
Direct secret flags are supported but can be visible in process listings; file
options are preferred. SSH login credentials, sudo credentials, and the worker
token remain separate.

## Worker Self-Monitoring

A worker must not run a monitor that represents its own process or host. The
control plane excludes it during selection; the worker also rejects a task that
names its own worker/host identity once those fields are signed into the task.
Peer health checks and control-plane selfcheck are separate from customer
monitor execution.

## Implementation Work Packages

1. [ ] Add signed account/config identity to `internal/wire` and runner state.
2. [ ] Validate task account and credential scope locally before dispatch.
3. [x] Implement in-memory reconnect token rotation without stopping worker
   subsystems.
4. [ ] Implement one-time bootstrap, durable token storage, and rotation
   acknowledgement.
5. [ ] Add worker disable/revoke behavior and visible stale-config state.
6. [ ] Add mTLS as an optional first transport, then require it for Raft clusters.
7. [ ] Add public-task grant and SSRF-safe executor only after private isolation is
   proven.

## Acceptance Tests

- [ ] A private worker never leases or executes another account's normal task.
- [ ] A forged capability or account field cannot widen scope.
- [ ] Invalid, expired, downgraded, or differently scoped signed config is rejected.
- [ ] Bootstrap tokens are single-use and absent from disk after exchange.
- [x] Rotation reconnects without stopping web, inventory, metrics, or cluster.
- [ ] Credential snapshots contain only the worker account and allowed methods.
- [ ] Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix
  sockets, or a private redirect target.
- [ ] Revocation prevents reconnect and clears in-memory credentials.