worker/docs/control-plane-protocol.md

# Control-Plane Protocol

## Authority

`internal/wire/types.go` is the executable schema. New fields must be optional
during rollout, and new branches require decode tests against both old and new
frames. Version strings are informational; explicit capabilities select
protocol features.

The worker connects to:

```text
GET /worker?token=<RSMON_TOKEN>
```

`/api/worker` and the HTTP jobs/results APIs remain compatibility paths. New
workers use WebSocket task envelopes.

Public endpoint and cluster identity requirements are defined in
[public-endpoint-and-identity.md](public-endpoint-and-identity.md).

## Frame Model

Every frame is a `wire.WorkerMessage` with `kind` and one active content
branch.

| Direction | `kind` | Active branch | State |
| --- | --- | --- | --- |
| Server to worker | `init` or `config` | `init` | Implemented |
| Server to worker | `task` | `task_envelope` | Implemented |
| Worker to server | `result` | `result` | Implemented |
| Worker to server | `result` | `notification_result` | Implemented |
| Worker to server | `result` | `server_metric` | Implemented |
| Worker to server | `heartbeat` | `heartbeat` | Implemented |
| Either | `error` | `error` | Implemented |
| Worker to server | `result` | inventory report | Planned |
| Server to worker | cluster config/witness | dedicated typed branch | Planned |
| Server to worker | diagnostic task | new task-envelope variant | Planned |

The legacy top-level `task` and `notification_task` branches are accepted for
rollout compatibility. A current `task_envelope` must activate exactly one
matching branch with a non-empty, matching inner and outer job ID and lease
token. Unsupported check kinds and attributable malformed envelopes produce
terminal reports; malformed envelopes without a usable identity are rejected
without execution or reporting.

## Initialization And Refresh

The worker is expected to propose `PUBLIC_URL` during registration and the
control plane to validate and canonicalize it; `wire.WorkerInit` returns the
accepted endpoint and supplies runtime values owned by the control plane:

- worker ID, region, advertised URL, capabilities, and concurrency;
- allowed notification methods and account IDs;
- optional linked server ID for host metrics;
- LLM endpoints;
- scoped notification credentials and system contacts;
- signed, cluster-scoped peer topology for selfcheck and Raft behavior.

Current worker behavior: the worker validates its local `PUBLIC_URL`
configuration at startup and *consumes* the accepted endpoint from
`wire.WorkerInit` (preferring `public_url`, falling back to the legacy `url`
field). Transmitting the proposed URL during registration is the pending RSMon
control-plane counterpart; the worker does not currently send it.

Worker ID, account, region, cluster, membership, role, topology generation, and
certificate identity are control-plane authority. Local environment or a peer
response cannot override them. Static peer environment remains lab-only.

The worker clamps supplied concurrency to its local maximum. Credentials are
replaced atomically in memory on refresh. Removed credentials must become
unavailable immediately after the refresh is applied.

`wire.WorkerInit` returns the accepted endpoint as `public_url`, with the
legacy `url` field still populated during the bounded migration; the worker
prefers `public_url` and ignores an unusable value (keeping the previous
accepted URL). `RegisterRequest.public_url` is the registration contract the
RSMon control-plane counterpart must populate when it wires worker-initiated
registration; the worker does not transmit it today. See
[public-endpoint-and-identity.md](public-endpoint-and-identity.md).

Private-worker hardening will add an immutable worker account ID, config
version, expiry, and signature. Until then the executable trusts the
authenticated control plane to send a correctly scoped config; server-side
selection remains the primary isolation boundary.

## Normal Task Envelope

`wire.TaskEnvelope` is a tagged union:

```json
{
  "kind": "task",
  "task_envelope": {
    "type": "check",
    "job_id": "uuid",
    "check": {
      "job_id": "uuid",
      "lease_token": "per-lease-secret",
      "check_id": 123,
      "monitor_id": 456,
      "kind": "http",
      "host": "example.com",
      "url": "https://example.com",
      "interval": 60,
      "settings": {}
    }
  }
}
```

`type=notification` activates `notification` instead. The worker enforces the
matching job IDs and non-empty lease token before dispatch. A task without one
recognized populated branch is rejected without execution; it is reported only
when the active branch provides attributable job and lease identity.

## Result Invariants

- Echo `job_id` and the exact `lease_token` from the task.
- Send one terminal result per execution attempt.
- Never retry a result by executing the task again. Result transport retries
  resend the same terminal report.
- Treat duplicate terminal acknowledgements as success.
- Do not infer task acceptance from a WebSocket write alone; durable ownership
  remains on the control plane until it validates the result.
- Bound error strings and provider responses before transmission.

## Host Metrics

`wire.ServerMetricReport` is sent only after `WorkerInit.ServerID` is present.
The worker collects locally and sends bounded snapshots. The control plane
validates worker/server/account ownership and persists both the latest cache
and VictoriaMetrics points. The worker does not have TSDB credentials.

## Protocol Work Packages

### P1: Conformance Tests

Files:

- `internal/wire/types_test.go`
- `internal/distworker/runner_protocol_test.go`

Implemented:

- strict single-branch envelope selection with matching job-ID and lease
  validation;
- unsupported-kind and attributable malformed-task terminal reporting;
- atomic config and credential replacement.

Still required:

- decode conformance coverage for every current frame branch and unknown
  optional fields;
- explicit drain behavior and stale-lease acknowledgement coverage.

### P2: Safe Token Rotation

`Runner.RotateToken` replaces the token in memory, closes only the active
control-plane WebSocket, and reconnects without terminating worker services.
The replacement is not persisted: a process restart still uses its configured
startup token. The control plane must tolerate connection-scoped result resend
using the leased job and lease token for idempotency.

Acceptance: rotating from the web console produces a reconnect using the new
token while the HTTP listener, collectors, and optional cluster stay running.

### P3: Signed Private-Worker Config

Add to the init/config branch:

- `account_id` for private workers;
- monotonic `config_version`;
- `issued_at` and `expires_at`;
- signature key ID and Ed25519 signature over canonical payload bytes.

Reject regressions, invalid signatures, expired config, and account changes.
Keep the last valid config only until its expiry; do not silently accept an
invalid replacement.

### P4: New Typed Branches

Inventory, diagnostic, and critical-cluster messages each receive a dedicated
wire type. Do not tunnel them through `event` or arbitrary `json.RawMessage`.
Each branch must define payload limits, account/target validation ownership,
idempotency, and compatibility behavior before implementation.