worker/docs/web-console-and-observability.md

# Web Console And Host Observability

## Current Implementation

`internal/webapp` is a functional embedded HTTP application backed by SQLite.
It provides authenticated overview, apps, checks, notifications, logs, status,
settings, updates, password rotation, token rotation, audit retention, and
cluster status endpoints. Static assets and templates are embedded in the
worker binary.

The listener defaults to `WORKER_HOST=0.0.0.0` and `WORKER_PORT=27401`.
Compose publishes it on host loopback by default through `WORKER_BIND_IP`; that
variable is a Compose interpolation setting, not a variable read by the binary.

## Authentication

Two modes are implemented:

| Configuration | Behavior |
| --- | --- |
| Both `WORKER_LOGIN` and `WORKER_PASSWORD` set | Basic-auth-backed operator sessions and `/web/api/*` access |
| Both empty | Local bcrypt user; a one-time password is printed on first start |
| Only one set | Startup error |

Sessions are HTTP-only and SameSite strict, with a 30-minute idle and 8-hour
absolute lifetime. State-changing browser requests require CSRF validation.
Authenticated responses use no-store and restrictive security headers.

OAuth/device authorization is not implemented. Public exposure requires a TLS
reverse proxy and network restrictions. The accepted target name is
`PUBLIC_URL`; `WORKER_URL` is current legacy configuration during migration.
Neither setting enables TLS; it only advertises the externally reachable URL.

## Known Runtime Gap

`webapp.New` constructs `Inventory` and `Metrics`, but `Server.Start` currently
does not start either collector and `Server.Close` does not stop them. As a
result, production `/apps` and `/status` pages can remain empty even though
collector unit tests pass.

This is the first required web-console change:

1. Start both collectors with the server context before accepting requests.
2. Stop and wait for them during `Close` and context cancellation.
3. Make collector start idempotent and collector failure observable.
4. Add a server integration test that observes populated inventory and metrics
   after startup.

## Inventory View

The current local collector reads Linux `/proc`, process command lines and
working directories, TCP listeners, and resource data, then stores snapshots
in SQLite. It does not yet provide complete grouping, Docker, Compose, systemd,
nginx, UDP, or deploymentd-compatible inventory. The implementation plan is in
[inventory.md](inventory.md).

## Host Metrics

There are two related collectors:

- `internal/distworker/server_metrics.go` sends the linked server's CPU,
  memory, disk, load, uptime, process, and network snapshot to the control
  plane every five seconds. This path is implemented.
- `internal/webapp/metrics.go` drives local status pages and a local ring. It
  exists but is affected by the lifecycle gap above.

The control-plane report uses interval rates for CPU and aggregate networking.
The local metrics implementation currently computes CPU from lifetime totals;
align it with interval deltas so local and remote views agree.

## Compose Management Target

Compose management remains planned and is opt-in. Implement it after read-only
inventory is complete, under a separate `internal/composeops` package.

Required constraints:

- fixed stacks root, default `/opt/stacks`, with canonical path containment;
- explicit `docker compose -f <validated-file>` invocation;
- no shell interpolation and no free-form host command endpoint;
- per-stack single-flight action lock;
- async `202 Accepted` operations with bounded progress streaming;
- viewer/operator/admin permissions and audit records;
- typed confirmation for down, delete, and restore;
- compose validation before save and backup before deploy;
- Docker socket treated as root-equivalent and disabled by default.

The first Compose release includes read/list/status/logs only. Mutation, file
editing, environment editing, terminal access, backup, and restore are separate
release gates rather than one large feature switch.

## Host Status Extensions

Implement in this order:

1. Correct `/proc`, `statfs`, and `/proc/net/dev` interval collection.
2. Add bounded local 24-hour one-minute aggregates.
3. Add optional `lsblk --json` device topology.
4. Add optional cached SMART and sensors data.
5. Add Docker daemon and per-container status only when the socket capability
   is enabled.

Missing commands or permissions produce unavailable sections, not worker
startup failure. All subprocesses use fixed argument arrays, deadlines, output
limits, and allowlisted executable names.

## Health And Updates

`GET /healthz` is local process liveness. The image healthcheck should call
this endpoint, not `rsmon-worker health`, because the latter probes the control
plane `/up` endpoint. Control-plane outage is a selfcheck/readiness event, not
proof that the worker process is dead.

In-place binary replacement from the web UI is not implemented and should not
run with ambient root access. The systemd update flow should download a signed
artifact to a staging path and require an external privileged installer or
explicit operator command to activate it.

## Acceptance Tests

- Starting `webapp.Server` produces nonempty metrics and process inventory on
  Linux and stops every collector cleanly.
- Both auth modes work, XOR credentials fail startup, CSRF protects writes,
  and unauthenticated API calls fail.
- A public bind is documented as insecure without TLS; no credential is logged.
- `/healthz` remains healthy during a simulated control-plane outage.
- Every host command has timeout/output-limit tests and rejects user-supplied
  executable or path traversal.
- Compose mutation cannot operate outside the configured stacks root.