worker/docs/control-plane-protocol.md

Control-Plane Protocol

Authority

internal/wire/types.go is the executable schema. New fields must be optional during rollout, and new branches require decode tests against both old and new frames. Version strings are informational; explicit capabilities select protocol features.

The worker connects to:

GET /worker?token=<RSMON_TOKEN>

/api/worker and the HTTP jobs/results APIs remain compatibility paths. New workers use WebSocket task envelopes.

Public endpoint and cluster identity requirements are defined in public-endpoint-and-identity.md.

Frame Model

Every frame is a wire.WorkerMessage with kind and one active content branch.

Direction	kind	Active branch	State
Server to worker	init or config	init	Implemented
Server to worker	task	task_envelope	Implemented
Worker to server	result	result	Implemented
Worker to server	result	notification_result	Implemented
Worker to server	result	server_metric	Implemented
Worker to server	heartbeat	heartbeat	Implemented
Either	error	error	Implemented
Worker to server	result	inventory report	Planned
Server to worker	cluster config/witness	dedicated typed branch	Planned
Server to worker	diagnostic task	new task-envelope variant	Planned

The legacy top-level task and notification_task branches are accepted for rollout compatibility. A current task_envelope must activate exactly one matching branch with a non-empty, matching inner and outer job ID and lease token. Unsupported check kinds and attributable malformed envelopes produce terminal reports; malformed envelopes without a usable identity are rejected without execution or reporting.

Initialization And Refresh

The worker proposes PUBLIC_URL during registration. The control plane validates and canonicalizes it; wire.WorkerInit returns the accepted endpoint and supplies runtime values owned by the control plane:

• worker ID, region, advertised URL, capabilities, and concurrency;
• allowed notification methods and account IDs;
• optional linked server ID for host metrics;
• LLM endpoints;
• scoped notification credentials and system contacts;
• signed, cluster-scoped peer topology for selfcheck and Raft behavior.

Worker ID, account, region, cluster, membership, role, topology generation, and certificate identity are control-plane authority. Local environment or a peer response cannot override them. Static peer environment remains lab-only.

The worker clamps supplied concurrency to its local maximum. Credentials are replaced atomically in memory on refresh. Removed credentials must become unavailable immediately after the refresh is applied.

Private-worker hardening will add an immutable worker account ID, config version, expiry, and signature. Until then the executable trusts the authenticated control plane to send a correctly scoped config; server-side selection remains the primary isolation boundary.

Normal Task Envelope

wire.TaskEnvelope is a tagged union:

{
  "kind": "task",
  "task_envelope": {
    "type": "check",
    "job_id": "uuid",
    "check": {
      "job_id": "uuid",
      "lease_token": "per-lease-secret",
      "check_id": 123,
      "monitor_id": 456,
      "kind": "http",
      "host": "example.com",
      "url": "https://example.com",
      "interval": 60,
      "settings": {}
    }
  }
}

type=notification activates notification instead. The worker enforces the matching job IDs and non-empty lease token before dispatch. A task without one recognized populated branch is rejected without execution; it is reported only when the active branch provides attributable job and lease identity.

Result Invariants

• Echo job_id and the exact lease_token from the task.
• Send one terminal result per execution attempt.
• Never retry a result by executing the task again. Result transport retries resend the same terminal report.
• Treat duplicate terminal acknowledgements as success.
• Do not infer task acceptance from a WebSocket write alone; durable ownership remains on the control plane until it validates the result.
• Bound error strings and provider responses before transmission.

Host Metrics

wire.ServerMetricReport is sent only after WorkerInit.ServerID is present. The worker collects locally and sends bounded snapshots. The control plane validates worker/server/account ownership and persists both the latest cache and VictoriaMetrics points. The worker does not have TSDB credentials.

Protocol Work Packages

P1: Conformance Tests

Files:

• internal/wire/types_test.go
• internal/distworker/runner_protocol_test.go

Implemented:

• strict single-branch envelope selection with matching job-ID and lease validation;
• unsupported-kind and attributable malformed-task terminal reporting;
• atomic config and credential replacement.

Still required:

• decode conformance coverage for every current frame branch and unknown optional fields;
• explicit drain behavior and stale-lease acknowledgement coverage.

P2: Safe Token Rotation

Runner.RotateToken replaces the token in memory, closes only the active control-plane WebSocket, and reconnects without terminating worker services. The replacement is not persisted: a process restart still uses its configured startup token. The control plane must tolerate connection-scoped result resend using the leased job and lease token for idempotency.

Acceptance: rotating from the web console produces a reconnect using the new token while the HTTP listener, collectors, and optional cluster stay running.

P3: Signed Private-Worker Config

Add to the init/config branch:

• account_id for private workers;
• monotonic config_version;
• issued_at and expires_at;
• signature key ID and Ed25519 signature over canonical payload bytes.

Reject regressions, invalid signatures, expired config, and account changes. Keep the last valid config only until its expiry; do not silently accept an invalid replacement.

P4: New Typed Branches

Inventory, diagnostic, and critical-cluster messages each receive a dedicated wire type. Do not tunnel them through event or arbitrary json.RawMessage. Each branch must define payload limits, account/target validation ownership, idempotency, and compatibility behavior before implementation.