worker/docs/tasks-and-notifications.md

Normal Tasks And Notifications

Normal Check Execution

Normal work is centrally scheduled. The control plane owns PostgreSQL task rows, selection, leases, retries, deadlines, and dead-letter state. The worker owns only one execution attempt and its terminal report.

The implemented path is:

1. internal/distworker/client.go opens the authenticated WebSocket.
2. internal/distworker/runner.go selects a recognized populated task branch.
3. A bounded worker pool executes the task under its deadline.
4. internal/checkexec/exec.go dispatches to a database-free check package.
5. The runner queues one result and the result writer returns it to the control plane with the exact lease token.

Implemented check kinds are HTTP, SSL, SSH, FTP, DNS, WHOIS, BSSL, LLM, LLM-HTTP, ping, TCP, and UDP. Control-plane selection uses capabilities. The local dispatcher supports only its explicit switch cases, but strict malformed envelope and unsupported-kind result handling remains work package N1 rather than a complete fail-closed protocol response.

Queue And Shutdown Rules

• Input and output queues are bounded. Backpressure must not create unbounded goroutines or memory use.
• Concurrency is supplied by the control plane but clamped by the local maximum.
• A task panic is recovered at the task boundary and reported as a failed attempt; it must not kill the runner.
• SIGTERM stops accepting tasks, allows bounded in-flight completion, attempts final result delivery, and then exits.
• WebSocket reconnect does not re-run an in-flight or completed task.
• HTTP polling remains compatibility-only and must not become a second normal scheduler.

Delegated Notifications

internal/distworker/notification.go executes one pre-rendered delivery attempt. It does not render alert policy and does not own retry scheduling.

Method	State	Credential source
Email	Implemented	scoped SMTP list from init/config
Telegram	Implemented	scoped bot list from init/config
Webhook	Implemented	current shared signing configuration
Mattermost	Implemented	current shared defaults/task endpoint
SMS	Explicitly unsupported	no provider selected
Voice	Explicitly unsupported	no provider selected

SMS and voice return permanent with unsupported_method; they must never be reported as delivered or silently dropped.

Worker result statuses are delivered, retryable, permanent, and partial. Transport failures, provider rate limits, and provider 5xx responses are retryable. Invalid recipients, authentication failures, malformed payloads, and unsupported methods are permanent. Positive provider retry-after values are included in the result; the control plane owns the actual retry time.

Credentials

Credentials arrive in WorkerInit.Credentials and remain in memory. SMTP and Telegram tasks may request a credential ID; the worker resolves only that ID from the authorized pushed set. Missing IDs are permanent failures. Do not fall back to another account's or platform credential.

The following fields must be redacted from logs and UI buffers: password, token, API key, signing secret, hook URL, authorization headers, and fields ending in _secret. Provider response text is bounded and must not include response bodies that could contain secrets.

Webhook and Mattermost need persisted per-account credentials in the control plane before they can claim the same isolation guarantees as SMTP and Telegram. The worker wire shape already separates method configuration; do not introduce environment-global fallback for private workers.

Selfcheck Notifications

System selfcheck alerts are not normal notification tasks. They allow a worker to notify configured system contacts when the control plane itself is unreachable. They use system contacts and credentials from config, local jitter, and local deduplication. They must not impersonate customer delivery or mutate control-plane message state.

The local notifications page currently records selfcheck deliveries. Normal delegated delivery attempts should be added to the same bounded view after redaction, with job ID, method, status, duration, and time only.

Implementation Work Packages

N1: Runner Integration Coverage

Add strict envelope/inner job-ID validation and structured unsupported-kind results. Add WebSocket integration tests for reconnect, task panic recovery, queue backpressure, result resend, stale lease behavior, malformed envelopes, unsupported kinds, and graceful drain. Target internal/distworker/runner_protocol_test.go and a local test WebSocket server.

N2: Complete Local Delivery Audit

Record delegated notification outcomes in the bounded local notification ring. Never store recipient values, rendered body, credentials, or full provider responses in SQLite. Verify /notifications shows both selfcheck and delegated attempts without leaking secrets.

N3: Per-Account Webhook And Mattermost Credentials

Once the control plane sends credential IDs and account-scoped records, require an exact credential match just like SMTP and Telegram. Remove global fallback for private workers. Add cross-account negative tests.

N4: SMS And Voice Providers

Select a provider and define a credential wire type before adding an executor. Implementation must include timeout, retry classification, provider response redaction, idempotency support, and contract tests. Until all are present, retain the explicit permanent unsupported result.

Acceptance Tests

• Every supported check kind can execute from a serialized task without a database connection.
• A stale or missing lease token cannot produce an accepted normal result.
• One task causes at most one local execution during reconnect and duplicate frame delivery.
• Private workers cannot resolve a credential outside their authorized set.
• SMTP 4xx/rate-limit conditions are retryable; invalid authentication is permanent.
• SMS and voice remain explicit failures until provider tests exist.
• Logs and local UI contain no pushed credential values.