worker/docs/tasks-and-notifications.md

# Normal Tasks And Notifications

## Normal Check Execution

Normal work is centrally scheduled. The control plane owns PostgreSQL task
rows, selection, leases, retries, deadlines, and dead-letter state. The worker
owns only one execution attempt and its terminal report.

The implemented path is:

1. `internal/distworker/client.go` opens the authenticated WebSocket.
2. `internal/distworker/runner.go` selects a recognized populated task branch.
3. A bounded worker pool executes the task under its deadline.
4. `internal/checkexec/exec.go` dispatches to a database-free check package.
5. The runner queues one result and the result writer returns it to the control
   plane with the exact lease token.

Implemented check kinds are HTTP, SSL, SSH, FTP, DNS, WHOIS, BSSL, LLM,
LLM-HTTP, ping, TCP, and UDP. Control-plane selection uses capabilities. The
runner and local executor share one supported-kind registry. Websocket task
envelopes require exactly one payload branch, matching non-empty outer/inner
job IDs, and a lease token. Unsupported checks return one terminal failed
result without execution. A safely attributable malformed single branch
returns a protocol failure; ambiguous and mismatched envelopes fail closed.

## Queue And Shutdown Rules

- Input and output queues are bounded. Backpressure must not create unbounded
  goroutines or memory use.
- Concurrency is supplied by the control plane but clamped by the local
  maximum.
- A task panic is recovered at the task boundary and reported as a failed
  attempt; it must not kill the runner.
- SIGTERM stops accepting new tasks and waits for active dispatchers before the
  execution pool closes. A bounded final-result drain across websocket shutdown
  is not implemented yet.
- WebSocket reconnect does not re-run an in-flight or completed task.
- HTTP polling remains compatibility-only and must not become a second normal
  scheduler.

## Delegated Notifications

`internal/distworker/notification.go` executes one pre-rendered delivery
attempt. It does not render alert policy and does not own retry scheduling.

| Method | State | Credential source |
| --- | --- | --- |
| Email | Implemented | scoped SMTP list from init/config |
| Telegram | Implemented | scoped bot list from init/config |
| Webhook | Implemented | current shared signing configuration |
| Mattermost | Implemented | current shared defaults/task endpoint |
| SMS | Explicitly unsupported | no provider selected |
| Voice | Explicitly unsupported | no provider selected |

SMS and voice return `permanent` with `unsupported_method`; they must never be
reported as delivered or silently dropped.

Worker result statuses are `delivered`, `retryable`, `permanent`, and
`partial`. Transport failures, provider rate limits, and provider 5xx responses
are retryable. Invalid recipients, authentication failures, malformed payloads,
and unsupported methods are permanent. Positive provider retry-after values
are included in the result; the control plane owns the actual retry time.

## Credentials

Credentials arrive in `WorkerInit.Credentials` and remain in memory. SMTP and
Telegram tasks may request a credential ID; the worker resolves only that ID
from the authorized pushed set. Missing IDs are permanent failures. Do not
fall back to another account's or platform credential.

The following fields must be redacted from logs and UI buffers: password,
token, API key, signing secret, hook URL, authorization headers, and fields
ending in `_secret`. Provider response text is bounded and must not include
response bodies that could contain secrets.

Webhook and Mattermost need persisted per-account credentials in the control
plane before they can claim the same isolation guarantees as SMTP and
Telegram. The worker wire shape already separates method configuration; do not
introduce environment-global fallback for private workers.

## Selfcheck Notifications

System selfcheck alerts are not normal notification tasks. They allow a worker
to notify configured system contacts when the control plane itself is
unreachable. They use system contacts and credentials from config, local
jitter, and local deduplication. They must not impersonate customer delivery or
mutate control-plane message state.

The local notifications page records both selfcheck and delegated delivery
attempts. Delegated rows contain job ID, method, status, duration, and time
only. Recipient, body, credentials, hook URL, authorization, provider response,
and error text are never copied into the bounded ring. Invalid deadlines and
recovered executor panics produce static permanent results and one redacted
local row.

## Implementation Work Packages

### N1: Runner Integration Coverage (partial)

Strict envelope/inner job-ID/lease validation and structured unsupported-kind
results are implemented with runner-boundary tests. Live WebSocket reconnect
and result-resend tests are implemented. Stale-lease acknowledgement and
bounded graceful final-drain tests remain.

### N2: Complete Local Delivery Audit (implemented)

Delegated notification outcomes are recorded in the bounded local notification
ring. `/notifications` renders selfcheck and delegated attempts without storing
recipient values, rendered body, credentials, provider responses, or errors.

### N3: Per-Account Webhook And Mattermost Credentials

Once the control plane sends credential IDs and account-scoped records, require
an exact credential match just like SMTP and Telegram. Remove global fallback
for private workers. Add cross-account negative tests.

### N4: SMS And Voice Providers

Select a provider and define a credential wire type before adding an executor.
Implementation must include timeout, retry classification, provider response
redaction, idempotency support, and contract tests. Until all are present,
retain the explicit permanent unsupported result.

## Acceptance Tests

- Every supported check kind can execute from a serialized task without a
  database connection.
- A stale or missing lease token cannot produce an accepted normal result.
- One task causes at most one local execution during reconnect and duplicate
  frame delivery.
- Private workers cannot resolve a credential outside their authorized set.
- SMTP 4xx/rate-limit conditions are retryable; invalid authentication is
  permanent.
- SMS and voice remain explicit failures until provider tests exist.
- Logs and local UI contain no pushed credential values.