fix(worker): harden control-plane lifecycle

- reconnect safely after token rotation and retry leased results
- reject malformed tasks and remove production cluster debug mutation
- validate environment files and require immutable container images

BREAKING CHANGE: Docker install, deploy, and Compose now require an
immutable repository@sha256 image reference.

docs/changelog.md

@@ -11,9 +11,10 @@
   through `known_hosts` or a pinned fingerprint.
 - Added `--token-file`, `--url`, `--api-key`, and `--no-start` deployment
   options.
-- Added optional `--docker` deployment using the prebuilt
-  `reg.rsxx.ru/rsmon/rsmon-worker:latest` image. Remote Docker deployment pulls
-  the image on the target host and does not depend on the local architecture.
+- Added optional `--docker` deployment using a prebuilt image. Docker install
+  and deploy now require an immutable `repository@sha256:...` reference before
+  any Docker or remote-host mutation; the former mutable `latest` default is no
+  longer accepted.
 - Simplified the default systemd service to `Type=simple`, `User=root`, and
   `Restart=on-failure`.
 - Reworked the legacy `scripts/install-systemd.sh` script as a compatibility
@@ -34,3 +35,27 @@
   execution, and production worker job reporting.
 - Published the changes as commit `3256dcd` (`feat: add worker install and
   deploy`) on `master`, triggering the Docker image workflow.
+
+### Task protocol and local audit hardening
+
+- Required one task-envelope branch, matching outer/inner job IDs, and a
+  non-empty lease token before local execution.
+- Added structured terminal failures for unsupported check kinds and safely
+  attributable malformed envelopes.
+- Recorded delegated notification outcomes in the bounded `/notifications`
+  view using only job ID, method, status, duration, and time.
+- Added static permanent handling for invalid deadlines and recovered
+  notification executor panics without retaining secret-bearing text.
+- Removed the critical-cluster test-config endpoint, CLI flag, environment
+  switch, and production helper; hardcoded config application is test-only.
+- Made runner token rotation connection-scoped and in-memory: it reconnects
+  without stopping web, inventory, metrics, or cluster subsystems. Durable token
+  storage, bootstrap exchange, rotation acknowledgement, and revocation remain
+  unimplemented.
+- Added bounded resend of dequeued check and notification result envelopes after
+  websocket reconnect; control-plane application remains at-least-once and must
+  deduplicate by leased job and lease token. Failed metric snapshots are dropped
+  and replaced by the next periodic tick, not replayed.
+- SIGTERM stops new dispatch and waits for active work, but stale-lease
+  acknowledgement, bounded graceful final-result drain, and duplicate-frame
+  coverage remain open.

docs/control-plane-protocol.md

@@ -35,11 +35,11 @@ branch.
 | Server to worker | diagnostic task | new task-envelope variant | Planned |
 
 The legacy top-level `task` and `notification_task` branches are accepted for
-rollout compatibility. If a frame includes both a valid current
-`task_envelope` and a legacy branch, the worker executes only
-`task_envelope`. The current runner ignores malformed or unsupported task
-branches and does not yet compare the envelope job ID with the inner job ID;
-strict rejection and reporting are P1 work below.
+rollout compatibility. A current `task_envelope` must activate exactly one
+matching branch with a non-empty, matching inner and outer job ID and lease
+token. Unsupported check kinds and attributable malformed envelopes produce
+terminal reports; malformed envelopes without a usable identity are rejected
+without execution or reporting.
 
 ## Initialization And Refresh
 
@@ -86,9 +86,10 @@ selection remains the primary isolation boundary.
 }
 ```
 
-`type=notification` activates `notification` instead. Job IDs in the envelope
-and active branch must match once P1 validation lands. Today a task without a
-recognized populated branch is ignored without execution.
+`type=notification` activates `notification` instead. The worker enforces the
+matching job IDs and non-empty lease token before dispatch. A task without one
+recognized populated branch is rejected without execution; it is reported only
+when the active branch provides attributable job and lease identity.
 
 ## Result Invariants
 
@@ -117,20 +118,26 @@ Files:
 - `internal/wire/types_test.go`
 - `internal/distworker/runner_protocol_test.go`
 
-Tests:
+Implemented:
 
-- decode every current frame branch;
-- prefer the current envelope over duplicate legacy branches;
-- reject mismatched envelope and inner job IDs;
-- preserve unknown optional fields during compatible rollout;
-- reject missing lease tokens before execution result submission;
-- atomically replace config and credentials.
+- strict single-branch envelope selection with matching job-ID and lease
+  validation;
+- unsupported-kind and attributable malformed-task terminal reporting;
+- atomic config and credential replacement.
+
+Still required:
+
+- decode conformance coverage for every current frame branch and unknown
+  optional fields;
+- explicit drain behavior and stale-lease acknowledgement coverage.
 
 ### P2: Safe Token Rotation
 
-Current `Runner.RotateToken` closes the runner stop channel. Replace this with a
-connection-scoped cancellation path that stores the new token, closes only the
-active WebSocket, and reconnects without terminating worker services.
+`Runner.RotateToken` replaces the token in memory, closes only the active
+control-plane WebSocket, and reconnects without terminating worker services.
+The replacement is not persisted: a process restart still uses its configured
+startup token. The control plane must tolerate connection-scoped result resend
+using the leased job and lease token for idempotency.
 
 Acceptance: rotating from the web console produces a reconnect using the new
 token while the HTTP listener, collectors, and optional cluster stay running.

docs/critical-check-cluster.md

@@ -11,8 +11,10 @@ It does not yet execute `distributed_critical` checks, verify signed config,
 evaluate observation/region/notification quorum, encrypt snapshots, deliver a
 commit-backed outbox, or consume an external witness report.
 
-The hardcoded test-config endpoint and startup flag are development-only and
-must stay disabled in production.
+The former hardcoded test-config endpoint, startup flag, environment switch,
+and production helper have been removed. Hardcoded config application now
+exists only as an unexported `_test.go` helper and is absent from production
+builds.
 
 ## Non-Negotiable Separation
 
@@ -163,16 +165,16 @@ but cannot commit or fabricate incidents.
 
 ## Ordered Implementation
 
-1. Remove production exposure of debug config application.
-2. Complete versioned FSM types, command validation, and deterministic tests.
-3. Add mTLS identity and safe one-claim cluster bootstrap.
-4. Add signed config and observer-set adoption.
-5. Add deterministic scheduler and checkexec bridge in shadow mode.
-6. Implement observation aggregation, incident policy, and idempotency.
-7. Implement encrypted snapshots and restore/migration tests.
-8. Add metadata outbox executor and failover-safe delivery.
-9. Add external witness, replay, metrics, and operational runbooks.
-10. Run synthetic 3/5-node fault campaigns before any customer check.
+1. [x] Remove production exposure of debug config application.
+2. [ ] Complete versioned FSM types, command validation, and deterministic tests.
+3. [ ] Add mTLS identity and safe one-claim cluster bootstrap.
+4. [ ] Add signed config and observer-set adoption.
+5. [ ] Add deterministic scheduler and checkexec bridge in shadow mode.
+6. [ ] Implement observation aggregation, incident policy, and idempotency.
+7. [ ] Implement encrypted snapshots and restore/migration tests.
+8. [ ] Add metadata outbox executor and failover-safe delivery.
+9. [ ] Add external witness, replay, metrics, and operational runbooks.
+10. [ ] Run synthetic 3/5-node fault campaigns before any customer check.
 
 ## Release Gates
 

docs/implementation-roadmap.md

@@ -28,12 +28,15 @@ Worker repository:
 - wire local inventory and metrics collector lifecycle into web server start
   and shutdown;
 - validate full HTTP config, including `WORKER_URL`, in main startup;
-- make token rotation reconnect rather than stop the runner;
+- [x] reconnect in memory on token rotation without stopping the runner;
+- [x] resend bounded check/notification results after websocket reconnect;
 - define process policy when the web listener exits unexpectedly;
-- add WebSocket reconnect, drain, duplicate frame, and backpressure tests.
+- add process-exit, bounded final-drain, duplicate-frame, and backpressure
+  coverage.
 
-Gate: collectors populate real pages, rotation preserves all subsystems, and
-SIGTERM leaves no listener, collector, task, or SQLite goroutine behind.
+Gate: collectors populate real pages, rotation preserves all subsystems, result
+resend remains bounded and idempotent, and SIGTERM leaves no listener,
+collector, task, or SQLite goroutine behind.
 
 ## R2: Protocol And Credential Hardening
 

docs/private-workers.md

@@ -55,19 +55,22 @@ Bootstrap replay, expiry, worker-ID mismatch, or account mismatch fails closed.
 
 ## Token Rotation
 
-Rotation is a two-token handoff:
+The current worker receives a replacement token in memory. In-memory reconnect
+rotation is implemented: the control plane
+invalidates the old token immediately, then `Runner.RotateToken` closes only
+the active control-plane connection and reconnects with the replacement token.
+It does not stop the runner or its web, inventory, metrics, or cluster
+subsystems. There is no durable token write, overlap, rollback, or rotation
+acknowledgement contract yet; persistent replacement-token failures can require
+operator action.
 
-1. Control plane issues a replacement token and keeps the old token valid for a
-   short bounded overlap.
-2. Worker atomically writes the replacement secret.
-3. Worker closes only the active control-plane connection and reconnects.
-4. Successful authentication acknowledges rotation; control plane revokes the
-   old token.
-5. Failure retains the old token until overlap expires and emits an operator
-   warning.
-
-The current `Runner.RotateToken` terminates the runner by closing its stop
-channel. This must be fixed before claiming unattended rotation.
+If a websocket write fails after the worker has dequeued a check or notification
+result, it resends that envelope after reconnect. Delivery is therefore
+at-least-once; the control plane must deduplicate by the leased job and lease
+token before applying a resent result. Failed server-metric snapshots are
+dropped because they have no idempotency key and the next periodic collection
+replaces them. Metric snapshots are bound to the active control connection;
+snapshots collected while disconnected or for an older connection are dropped.
 
 ## Runtime Config Authentication
 
@@ -126,7 +129,8 @@ The standalone binary provides two systemd installation paths:
   installs a systemd-managed prebuilt image instead;
 - `rsmon-worker deploy` verifies an SSH host key, uploads the binary and a
   temporary mode-0600 environment file, and invokes `install` remotely;
-  `--docker` uploads only the environment and unit, then pulls on the target.
+  `--docker` uploads only the environment and unit, then pulls an explicitly
+  supplied immutable image digest on the target.
 
 Both default to `https://rsmon.ru` and accept `--token-file` for automation.
 Direct secret flags are supported but can be visible in process listings; file
@@ -143,22 +147,25 @@ monitor execution.
 
 ## Implementation Work Packages
 
-1. Add signed account/config identity to `internal/wire` and runner state.
-2. Validate task account and credential scope locally before dispatch.
-3. Implement one-time bootstrap and atomic token storage/rotation.
-4. Add worker disable/revoke behavior and visible stale-config state.
-5. Add mTLS as an optional first transport, then require it for Raft clusters.
-6. Add public-task grant and SSRF-safe executor only after private isolation is
+1. [ ] Add signed account/config identity to `internal/wire` and runner state.
+2. [ ] Validate task account and credential scope locally before dispatch.
+3. [x] Implement in-memory reconnect token rotation without stopping worker
+   subsystems.
+4. [ ] Implement one-time bootstrap, durable token storage, and rotation
+   acknowledgement.
+5. [ ] Add worker disable/revoke behavior and visible stale-config state.
+6. [ ] Add mTLS as an optional first transport, then require it for Raft clusters.
+7. [ ] Add public-task grant and SSRF-safe executor only after private isolation is
    proven.
 
 ## Acceptance Tests
 
-- A private worker never leases or executes another account's normal task.
-- A forged capability or account field cannot widen scope.
-- Invalid, expired, downgraded, or differently scoped signed config is rejected.
-- Bootstrap tokens are single-use and absent from disk after exchange.
-- Rotation reconnects without stopping web, inventory, metrics, or cluster.
-- Credential snapshots contain only the worker account and allowed methods.
-- Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix
+- [ ] A private worker never leases or executes another account's normal task.
+- [ ] A forged capability or account field cannot widen scope.
+- [ ] Invalid, expired, downgraded, or differently scoped signed config is rejected.
+- [ ] Bootstrap tokens are single-use and absent from disk after exchange.
+- [x] Rotation reconnects without stopping web, inventory, metrics, or cluster.
+- [ ] Credential snapshots contain only the worker account and allowed methods.
+- [ ] Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix
   sockets, or a private redirect target.
-- Revocation prevents reconnect and clears in-memory credentials.
+- [ ] Revocation prevents reconnect and clears in-memory credentials.

docs/sessions/2026-07-19-standalone-deployment.md

@@ -29,8 +29,9 @@ default and offer the prebuilt Docker image as an explicit alternative.
 
 - The default service uses `Type=simple`, runs as `root`, and restarts only on
   failure.
-- Docker deployment pulls `reg.rsxx.ru/rsmon/rsmon-worker:latest` on the target
-  host instead of uploading a local image or binary.
+- Docker deployment pulls an explicitly selected immutable image digest on the
+  target host instead of uploading a local image or binary. The original
+  mutable `latest` default was superseded by a supply-chain hardening change.
 - The existing source-available license remains unchanged until public-release
   licensing is decided.
 - Worker credentials are supplied at deployment time and are not stored in the

docs/tasks-and-notifications.md

@@ -17,9 +17,11 @@ The implemented path is:
 
 Implemented check kinds are HTTP, SSL, SSH, FTP, DNS, WHOIS, BSSL, LLM,
 LLM-HTTP, ping, TCP, and UDP. Control-plane selection uses capabilities. The
-local dispatcher supports only its explicit switch cases, but strict malformed
-envelope and unsupported-kind result handling remains work package N1 rather
-than a complete fail-closed protocol response.
+runner and local executor share one supported-kind registry. Websocket task
+envelopes require exactly one payload branch, matching non-empty outer/inner
+job IDs, and a lease token. Unsupported checks return one terminal failed
+result without execution. A safely attributable malformed single branch
+returns a protocol failure; ambiguous and mismatched envelopes fail closed.
 
 ## Queue And Shutdown Rules
 
@@ -29,8 +31,9 @@ than a complete fail-closed protocol response.
   maximum.
 - A task panic is recovered at the task boundary and reported as a failed
   attempt; it must not kill the runner.
-- SIGTERM stops accepting tasks, allows bounded in-flight completion, attempts
-  final result delivery, and then exits.
+- SIGTERM stops accepting new tasks and waits for active dispatchers before the
+  execution pool closes. A bounded final-result drain across websocket shutdown
+  is not implemented yet.
 - WebSocket reconnect does not re-run an in-flight or completed task.
 - HTTP polling remains compatibility-only and must not become a second normal
   scheduler.
@@ -83,26 +86,27 @@ unreachable. They use system contacts and credentials from config, local
 jitter, and local deduplication. They must not impersonate customer delivery or
 mutate control-plane message state.
 
-The local notifications page currently records selfcheck deliveries. Normal
-delegated delivery attempts should be added to the same bounded view after
-redaction, with job ID, method, status, duration, and time only.
+The local notifications page records both selfcheck and delegated delivery
+attempts. Delegated rows contain job ID, method, status, duration, and time
+only. Recipient, body, credentials, hook URL, authorization, provider response,
+and error text are never copied into the bounded ring. Invalid deadlines and
+recovered executor panics produce static permanent results and one redacted
+local row.
 
 ## Implementation Work Packages
 
-### N1: Runner Integration Coverage
+### N1: Runner Integration Coverage (partial)
 
-Add strict envelope/inner job-ID validation and structured unsupported-kind
-results. Add WebSocket integration tests for reconnect, task panic recovery,
-queue backpressure, result resend, stale lease behavior, malformed envelopes,
-unsupported kinds, and graceful drain. Target
-`internal/distworker/runner_protocol_test.go` and a local test WebSocket server.
+Strict envelope/inner job-ID/lease validation and structured unsupported-kind
+results are implemented with runner-boundary tests. Live WebSocket reconnect
+and result-resend tests are implemented. Stale-lease acknowledgement and
+bounded graceful final-drain tests remain.
 
-### N2: Complete Local Delivery Audit
+### N2: Complete Local Delivery Audit (implemented)
 
-Record delegated notification outcomes in the bounded local notification ring.
-Never store recipient values, rendered body, credentials, or full provider
-responses in SQLite. Verify `/notifications` shows both selfcheck and delegated
-attempts without leaking secrets.
+Delegated notification outcomes are recorded in the bounded local notification
+ring. `/notifications` renders selfcheck and delegated attempts without storing
+recipient values, rendered body, credentials, provider responses, or errors.
 
 ### N3: Per-Account Webhook And Mattermost Credentials