worker/docs/control-plane-protocol.md

# Control-Plane Protocol

## Authority

`internal/wire/types.go` is the executable schema. New fields must be optional
during rollout, and new branches require decode tests against both old and new
frames. Version strings are informational; explicit capabilities select
protocol features.

The worker connects to:

```text
GET /worker?token=<RSMON_TOKEN>
```

`/api/worker` and the HTTP jobs/results APIs remain compatibility paths. New
workers use WebSocket task envelopes.

## Frame Model

Every frame is a `wire.WorkerMessage` with `kind` and one active content
branch.

| Direction | `kind` | Active branch | State |
| --- | --- | --- | --- |
| Server to worker | `init` or `config` | `init` | Implemented |
| Server to worker | `task` | `task_envelope` | Implemented |
| Worker to server | `result` | `result` | Implemented |
| Worker to server | `result` | `notification_result` | Implemented |
| Worker to server | `result` | `server_metric` | Implemented |
| Worker to server | `heartbeat` | `heartbeat` | Implemented |
| Either | `error` | `error` | Implemented |
| Worker to server | `result` | inventory report | Planned |
| Server to worker | cluster config/witness | dedicated typed branch | Planned |
| Server to worker | diagnostic task | new task-envelope variant | Planned |

The legacy top-level `task` and `notification_task` branches are accepted for
rollout compatibility. A current `task_envelope` must activate exactly one
matching branch with a non-empty, matching inner and outer job ID and lease
token. Unsupported check kinds and attributable malformed envelopes produce
terminal reports; malformed envelopes without a usable identity are rejected
without execution or reporting.

## Initialization And Refresh

`wire.WorkerInit` supplies runtime values owned by the control plane:

- worker ID, region, advertised URL, capabilities, and concurrency;
- allowed notification methods and account IDs;
- optional linked server ID for host metrics;
- LLM endpoints;
- scoped notification credentials and system contacts;
- peer workers for selfcheck/cluster-adjacent behavior.

The worker clamps supplied concurrency to its local maximum. Credentials are
replaced atomically in memory on refresh. Removed credentials must become
unavailable immediately after the refresh is applied.

Private-worker hardening will add an immutable worker account ID, config
version, expiry, and signature. Until then the executable trusts the
authenticated control plane to send a correctly scoped config; server-side
selection remains the primary isolation boundary.

## Normal Task Envelope

`wire.TaskEnvelope` is a tagged union:

```json
{
  "kind": "task",
  "task_envelope": {
    "type": "check",
    "job_id": "uuid",
    "check": {
      "job_id": "uuid",
      "lease_token": "per-lease-secret",
      "check_id": 123,
      "monitor_id": 456,
      "kind": "http",
      "host": "example.com",
      "url": "https://example.com",
      "interval": 60,
      "settings": {}
    }
  }
}
```

`type=notification` activates `notification` instead. The worker enforces the
matching job IDs and non-empty lease token before dispatch. A task without one
recognized populated branch is rejected without execution; it is reported only
when the active branch provides attributable job and lease identity.

## Result Invariants

- Echo `job_id` and the exact `lease_token` from the task.
- Send one terminal result per execution attempt.
- Never retry a result by executing the task again. Result transport retries
  resend the same terminal report.
- Treat duplicate terminal acknowledgements as success.
- Do not infer task acceptance from a WebSocket write alone; durable ownership
  remains on the control plane until it validates the result.
- Bound error strings and provider responses before transmission.

## Host Metrics

`wire.ServerMetricReport` is sent only after `WorkerInit.ServerID` is present.
The worker collects locally and sends bounded snapshots. The control plane
validates worker/server/account ownership and persists both the latest cache
and VictoriaMetrics points. The worker does not have TSDB credentials.

## Protocol Work Packages

### P1: Conformance Tests

Files:

- `internal/wire/types_test.go`
- `internal/distworker/runner_protocol_test.go`

Implemented:

- strict single-branch envelope selection with matching job-ID and lease
  validation;
- unsupported-kind and attributable malformed-task terminal reporting;
- atomic config and credential replacement.

Still required:

- decode conformance coverage for every current frame branch and unknown
  optional fields;
- explicit drain behavior and stale-lease acknowledgement coverage.

### P2: Safe Token Rotation

`Runner.RotateToken` replaces the token in memory, closes only the active
control-plane WebSocket, and reconnects without terminating worker services.
The replacement is not persisted: a process restart still uses its configured
startup token. The control plane must tolerate connection-scoped result resend
using the leased job and lease token for idempotency.

Acceptance: rotating from the web console produces a reconnect using the new
token while the HTTP listener, collectors, and optional cluster stay running.

### P3: Signed Private-Worker Config

Add to the init/config branch:

- `account_id` for private workers;
- monotonic `config_version`;
- `issued_at` and `expires_at`;
- signature key ID and Ed25519 signature over canonical payload bytes.

Reject regressions, invalid signatures, expired config, and account changes.
Keep the last valid config only until its expiry; do not silently accept an
invalid replacement.

### P4: New Typed Branches

Inventory, diagnostic, and critical-cluster messages each receive a dedicated
wire type. Do not tunnel them through `event` or arbitrary `json.RawMessage`.
Each branch must define payload limits, account/target validation ownership,
idempotency, and compatibility behavior before implementation.