fix(worker): harden control-plane lifecycle
Все проверки выполнены успешно
CI / test (push) Successful in 2m32s
Docker / Build and publish worker image (push) Successful in 18m17s

- reconnect safely after token rotation and retry leased results
- reject malformed tasks and remove production cluster debug mutation
- validate environment files and require immutable container images

BREAKING CHANGE: Docker install, deploy, and Compose now require an
immutable repository@sha256 image reference.
Этот коммит содержится в:
Gleb Tv
2026-07-19 23:11:43 +03:00
родитель 6937674449
Коммит e987f24903
38 изменённых файлов: 2203 добавлений и 674 удалений

Просмотреть файл

@@ -17,9 +17,11 @@ The implemented path is:
Implemented check kinds are HTTP, SSL, SSH, FTP, DNS, WHOIS, BSSL, LLM,
LLM-HTTP, ping, TCP, and UDP. Control-plane selection uses capabilities. The
local dispatcher supports only its explicit switch cases, but strict malformed
envelope and unsupported-kind result handling remains work package N1 rather
than a complete fail-closed protocol response.
runner and local executor share one supported-kind registry. Websocket task
envelopes require exactly one payload branch, matching non-empty outer/inner
job IDs, and a lease token. Unsupported checks return one terminal failed
result without execution. A safely attributable malformed single branch
returns a protocol failure; ambiguous and mismatched envelopes fail closed.
## Queue And Shutdown Rules
@@ -29,8 +31,9 @@ than a complete fail-closed protocol response.
maximum.
- A task panic is recovered at the task boundary and reported as a failed
attempt; it must not kill the runner.
- SIGTERM stops accepting tasks, allows bounded in-flight completion, attempts
final result delivery, and then exits.
- SIGTERM stops accepting new tasks and waits for active dispatchers before the
execution pool closes. A bounded final-result drain across websocket shutdown
is not implemented yet.
- WebSocket reconnect does not re-run an in-flight or completed task.
- HTTP polling remains compatibility-only and must not become a second normal
scheduler.
@@ -83,26 +86,27 @@ unreachable. They use system contacts and credentials from config, local
jitter, and local deduplication. They must not impersonate customer delivery or
mutate control-plane message state.
The local notifications page currently records selfcheck deliveries. Normal
delegated delivery attempts should be added to the same bounded view after
redaction, with job ID, method, status, duration, and time only.
The local notifications page records both selfcheck and delegated delivery
attempts. Delegated rows contain job ID, method, status, duration, and time
only. Recipient, body, credentials, hook URL, authorization, provider response,
and error text are never copied into the bounded ring. Invalid deadlines and
recovered executor panics produce static permanent results and one redacted
local row.
## Implementation Work Packages
### N1: Runner Integration Coverage
### N1: Runner Integration Coverage (partial)
Add strict envelope/inner job-ID validation and structured unsupported-kind
results. Add WebSocket integration tests for reconnect, task panic recovery,
queue backpressure, result resend, stale lease behavior, malformed envelopes,
unsupported kinds, and graceful drain. Target
`internal/distworker/runner_protocol_test.go` and a local test WebSocket server.
Strict envelope/inner job-ID/lease validation and structured unsupported-kind
results are implemented with runner-boundary tests. Live WebSocket reconnect
and result-resend tests are implemented. Stale-lease acknowledgement and
bounded graceful final-drain tests remain.
### N2: Complete Local Delivery Audit
### N2: Complete Local Delivery Audit (implemented)
Record delegated notification outcomes in the bounded local notification ring.
Never store recipient values, rendered body, credentials, or full provider
responses in SQLite. Verify `/notifications` shows both selfcheck and delegated
attempts without leaking secrets.
Delegated notification outcomes are recorded in the bounded local notification
ring. `/notifications` renders selfcheck and delegated attempts without storing
recipient values, rendered body, credentials, provider responses, or errors.
### N3: Per-Account Webhook And Mattermost Credentials