fix(worker): harden control-plane lifecycle
Все проверки выполнены успешно
CI / test (push) Successful in 2m32s
Docker / Build and publish worker image (push) Successful in 18m17s
Все проверки выполнены успешно
CI / test (push) Successful in 2m32s
Docker / Build and publish worker image (push) Successful in 18m17s
- reconnect safely after token rotation and retry leased results - reject malformed tasks and remove production cluster debug mutation - validate environment files and require immutable container images BREAKING CHANGE: Docker install, deploy, and Compose now require an immutable repository@sha256 image reference.
Этот коммит содержится в:
@@ -55,19 +55,22 @@ Bootstrap replay, expiry, worker-ID mismatch, or account mismatch fails closed.
|
||||
|
||||
## Token Rotation
|
||||
|
||||
Rotation is a two-token handoff:
|
||||
The current worker receives a replacement token in memory. In-memory reconnect
|
||||
rotation is implemented: the control plane
|
||||
invalidates the old token immediately, then `Runner.RotateToken` closes only
|
||||
the active control-plane connection and reconnects with the replacement token.
|
||||
It does not stop the runner or its web, inventory, metrics, or cluster
|
||||
subsystems. There is no durable token write, overlap, rollback, or rotation
|
||||
acknowledgement contract yet; persistent replacement-token failures can require
|
||||
operator action.
|
||||
|
||||
1. Control plane issues a replacement token and keeps the old token valid for a
|
||||
short bounded overlap.
|
||||
2. Worker atomically writes the replacement secret.
|
||||
3. Worker closes only the active control-plane connection and reconnects.
|
||||
4. Successful authentication acknowledges rotation; control plane revokes the
|
||||
old token.
|
||||
5. Failure retains the old token until overlap expires and emits an operator
|
||||
warning.
|
||||
|
||||
The current `Runner.RotateToken` terminates the runner by closing its stop
|
||||
channel. This must be fixed before claiming unattended rotation.
|
||||
If a websocket write fails after the worker has dequeued a check or notification
|
||||
result, it resends that envelope after reconnect. Delivery is therefore
|
||||
at-least-once; the control plane must deduplicate by the leased job and lease
|
||||
token before applying a resent result. Failed server-metric snapshots are
|
||||
dropped because they have no idempotency key and the next periodic collection
|
||||
replaces them. Metric snapshots are bound to the active control connection;
|
||||
snapshots collected while disconnected or for an older connection are dropped.
|
||||
|
||||
## Runtime Config Authentication
|
||||
|
||||
@@ -126,7 +129,8 @@ The standalone binary provides two systemd installation paths:
|
||||
installs a systemd-managed prebuilt image instead;
|
||||
- `rsmon-worker deploy` verifies an SSH host key, uploads the binary and a
|
||||
temporary mode-0600 environment file, and invokes `install` remotely;
|
||||
`--docker` uploads only the environment and unit, then pulls on the target.
|
||||
`--docker` uploads only the environment and unit, then pulls an explicitly
|
||||
supplied immutable image digest on the target.
|
||||
|
||||
Both default to `https://rsmon.ru` and accept `--token-file` for automation.
|
||||
Direct secret flags are supported but can be visible in process listings; file
|
||||
@@ -143,22 +147,25 @@ monitor execution.
|
||||
|
||||
## Implementation Work Packages
|
||||
|
||||
1. Add signed account/config identity to `internal/wire` and runner state.
|
||||
2. Validate task account and credential scope locally before dispatch.
|
||||
3. Implement one-time bootstrap and atomic token storage/rotation.
|
||||
4. Add worker disable/revoke behavior and visible stale-config state.
|
||||
5. Add mTLS as an optional first transport, then require it for Raft clusters.
|
||||
6. Add public-task grant and SSRF-safe executor only after private isolation is
|
||||
1. [ ] Add signed account/config identity to `internal/wire` and runner state.
|
||||
2. [ ] Validate task account and credential scope locally before dispatch.
|
||||
3. [x] Implement in-memory reconnect token rotation without stopping worker
|
||||
subsystems.
|
||||
4. [ ] Implement one-time bootstrap, durable token storage, and rotation
|
||||
acknowledgement.
|
||||
5. [ ] Add worker disable/revoke behavior and visible stale-config state.
|
||||
6. [ ] Add mTLS as an optional first transport, then require it for Raft clusters.
|
||||
7. [ ] Add public-task grant and SSRF-safe executor only after private isolation is
|
||||
proven.
|
||||
|
||||
## Acceptance Tests
|
||||
|
||||
- A private worker never leases or executes another account's normal task.
|
||||
- A forged capability or account field cannot widen scope.
|
||||
- Invalid, expired, downgraded, or differently scoped signed config is rejected.
|
||||
- Bootstrap tokens are single-use and absent from disk after exchange.
|
||||
- Rotation reconnects without stopping web, inventory, metrics, or cluster.
|
||||
- Credential snapshots contain only the worker account and allowed methods.
|
||||
- Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix
|
||||
- [ ] A private worker never leases or executes another account's normal task.
|
||||
- [ ] A forged capability or account field cannot widen scope.
|
||||
- [ ] Invalid, expired, downgraded, or differently scoped signed config is rejected.
|
||||
- [ ] Bootstrap tokens are single-use and absent from disk after exchange.
|
||||
- [x] Rotation reconnects without stopping web, inventory, metrics, or cluster.
|
||||
- [ ] Credential snapshots contain only the worker account and allowed methods.
|
||||
- [ ] Public execution cannot reach loopback, link-local, RFC1918, metadata, Unix
|
||||
sockets, or a private redirect target.
|
||||
- Revocation prevents reconnect and clears in-memory credentials.
|
||||
- [ ] Revocation prevents reconnect and clears in-memory credentials.
|
||||
|
||||
Ссылка в новой задаче
Block a user