worker/docs/changelog.md

Changelog

2026-08-13

Source-install E2E in CI (work package 5)

• Added .github/workflows/test-ssh.yml, a Gitea Actions workflow that runs the Alpine/Ubuntu/Arch Docker/OpenSSH source-install matrix (make test-ssh) on pushes to master and on manual workflow_dispatch only, separate from the Docker-free unit CI. It does not trigger on pull_request: the distro fixtures execute the checked-out code inside privileged Docker, so untrusted PR code must never run there automatically. It bounds the job with timeout-minutes: 90 (the go-test -timeout 60m stays in place), scopes concurrency per ref (test-ssh-${{ gitea.ref }}), declares permissions: contents: read, uploads no artifacts, and cleans up on every path.
• Pinned the two actions to immutable full commit SHAs (verified against the GitHub API): actions/checkout@v4 -> 11d5960a326750d5838078e36cf38b85af677262 and actions/setup-go@v5 -> 40f1582b2485089dde7abd97c1529aa768e1baff. The repo-wide convention still leaves ci.yml/docker.yml on moving tags (accepted, documented risk); see docs/source-installation.md.
• Wired RSMON_TEST_IMAGE_ALPINE / RSMON_TEST_IMAGE_UBUNTU / RSMON_TEST_IMAGE_ARCH and RSMON_TEST_DOCKER_DNS repository variables (all empty by default) so CI can pin per-fixture mirror/snapshot images and a resolver for flaky CI DNS.
• Added scripts/ci/test-ssh.sh: preflights Docker and the harness's loopback port-publishing requirement with a tiny docker run -p 127.0.0.1::22 probe (fails fast with an actionable message plus diagnostics/fix options instead of a 60m timeout on an unsupported runner), then runs make test-ssh and traps EXIT to remove every leftover rsmon-worker-test-* container/network/image tag. Cleanup filters are anchored to the harness's own prefix/repository so they never touch a shared base image. The workflow adds an if: always() cleanup step as a belt-and-suspenders so a killed job never leaves test material on the runner.
• External network is fetched live by design (go.dev toolchain, rocketgit.ru source clone, distro repos); operators can pin a resolver via the RSMON_TEST_DOCKER_DNS repository variable (comma-separated nameservers, applied as docker run --dns ...) and mirror/snapshot overrides via RSMON_TEST_IMAGE_ALPINE / RSMON_TEST_IMAGE_UBUNTU / RSMON_TEST_IMAGE_ARCH.
• Fixed a history-dependent test fragility: TestSourceInstallDirtyCheckoutPreservesStaging and the rollback test's build-failure step dirty the tracked tree by appending a marker line to Makefile instead of git checkout master~1 -- Makefile, which silently stopped dirtying the tree once the last commit did not touch that file.

2026-08-12

Source-install hardening review

• Fail-closed remote scripts: checkout, branch resolution, and build steps now run under set -eu (and package/record steps chain with &&), so a failed checkout or fetch can never be masked by a stale rev-parse or subsequent command. The checkout step additionally refuses (git diff --quiet / --cached --quiet) before the destructive checkout -B, because -B silently discards local changes and would otherwise never fail on a dirty tree. A dirty-tree checkout failure surfaces as a check out branch error before the build runs; the new TestSourceInstallSSHCheckoutFailureNotMasked unit test and TestSourceInstallDirtyCheckoutPreservesStaging Docker test prove the previous staging binary and commit record are preserved byte-for-byte.
• Atomic toolchain replacement: the Go toolchain is downloaded, SHA-256 verified, extracted into a same-filesystem staging dir, verified to report the target version, and only then swapped into ToolchainDir with the prior toolchain moved to a sibling .go-backup that is restored on swap failure. A failed download/verify/extract/swap never destroys the prior Go.
• Record-after-build pairing: rsmon-worker.commit is written only after a successful build, so the record and the staged binary always correspond to the same commit. The build verifies <stage>.new --version before an atomic mv -f over the previous staging binary; GOMODCACHE is now set alongside GOCACHE inside the build dir so reruns reuse both caches.
• Origin verification: an existing checkout's remote.origin.url must exactly match the configured repository before anything is fetched or built.
• Repository hardening: only https:// clone URLs without userinfo are accepted (ValidateRepoURL, enforced before dialing and again when planning).
• Explicit charset validation for Go version and architecture overrides (sshinstall.ValidGoVersion / ValidGoArch) before any remote mutation.
• Bounded remote execution: each remote command is capped by --session-timeout (default 30m) and captured stdout is size-bounded alongside the existing stderr bound; deploy's streaming runRemote keeps its historical no-timeout behavior.
• The source installer now defaults to the remote's default branch (the public repo publishes master) instead of the plan's stale main default, while --branch still pins an explicit branch that must exist remotely. The README quickstart no longer shows the incorrect --branch main.
• CLI secret flags keep their compatibility, but docs now explicitly state that file options (-password-file, etc.) keep secrets out of argv and shell history while direct flags expose them through the process list.
• The harness accepts RSMON_TEST_DOCKER_DNS (comma-separated) to pin docker run --dns for fixture containers, so internet-facing installs are not at the mercy of a flaky local resolver.

Remote source-install execution (work package 3)

• Added installer.SourceInstall (internal/installer/sourceinstall.go): executes the source-install flow through the existing SSH transport, reusing the deploy command's SSHOptions (keys, passphrases, passwords, sudo passwords, known-hosts, pinned fingerprints) and its privilege path. Extracted the shared SSHOptions struct and a sudoWrap helper so deploy and source install cannot diverge.
• Steps implemented: minimal package-prerequisite install per distro (apk/apt/pacman/dnf, never a compiler), SHA-256-verified Go 1.26 toolchain download/extraction with an idempotent version-skip and temp-dir cleanup, clone-or-update of the public repository (with a bounded 3-attempt retry for transient DNS/TLS/proxy failures), resolution of the remote default branch (a pinned branch must exist remotely), a resolved branch and commit record at <BuildDir>/rsmon-worker.commit, and a staging build (CGO_ENABLED=0, -trimpath, repository -ldflags) verified via --version. The running service, config, and data directory are untouched (work package 4 boundary).
• Security: every interpolated remote value is single-quoted; branch and commit values are strictly validated; no worker token or control-plane credential is sent; sudo passwords travel only over session stdin; remote errors are bounded (stderr truncated in runRemoteOutput).
• Added unit tests for the remote scripts, option validation, branch/commit parsing, sudo wrapping, the secrets-absent contract, and an in-process real-SSH orchestration flow (with missing-pinned-branch, build-failure, and detection-failure paths).
• Added TestSourceInstallFixtures to the Docker/OpenSSH harness: each of Alpine, Ubuntu, and Arch installs from a clean state through the real harness transport (prerequisite install, verified Go 1.26, clone, resolved commit, staging build), then a rerun proves idempotency (same branch, toolchain reuse, no temp leaks). All three resolved the public repo's master at 4651deb2... in the recorded run. make test-ssh timeout raised to 60m.
• Documented the branch-resolution reality: the public repository currently publishes master, and the installer records whatever the remote default branch resolves to.

Source-install harness and planning foundations (work packages 1-2)

• Added internal/installer/harness: a reusable Docker/OpenSSH test harness that builds real OpenSSH containers for Alpine, Ubuntu, and Arch, waits for real network SSH readiness, captures the server host key into a temp known_hosts file, and tears the container, network, per-instance fixture image tag, and temp dir down reliably. It uses the golang.org/x/crypto/ssh library and known_hosts verification semantics the installer's deploy path relies on (the harness owns its connection code rather than reusing the installer functions) and never mocks SSH. A fresh known_hosts file trusts the first key (TOFU); the host-key mismatch test proves a different key is rejected before any command runs.
• Added distro fixtures under internal/installer/harness/testdata/fixtures. Alpine defaults to the reg.rsxx.ru/library/alpine:3 mirror; Ubuntu and Arch fall back to Docker Hub refs overridable via RSMON_TEST_IMAGE_<NAME>. Each fixture starts clean (no Go, no worker source) and authenticates with a bundled test key; password auth is disabled. The test key is strictly test-only - it grants root only to the disposable fixture containers - and must never be used outside the harness.
• Added opt-in integration controls: the Docker tests run only with RSMON_TEST_DOCKER=1 (make test-ssh); default make test and go test ./... skip them and never pull or start containers. make test also pins RSMON_TEST_DOCKER=0 so an exported opt-in flag cannot leak into the unit run.
• Integration tests assert real SSH round trips, clean target state, distro / package-manager / init detection per fixture, a full source plan including the pinned Go toolchain, host-key mismatch rejection, host-key stability, failed-start cleanup, and complete teardown (container, network, fixture image tag, and temp dir gone).
• Added internal/sshinstall: pure, unit-tested detection and planning for the source installer - os-release parsing, distro/package-manager/init resolution, uname -m to Go archive mapping, pinned Go 1.26 toolchain with published SHA-256, and a reviewable ordered plan. No remote execution yet.
• make test now includes the new packages; make test-ssh runs the live fixture matrix.

Public endpoint configuration (milestone 1 of public-endpoint-and-identity)

• PUBLIC_URL is now the canonical advertised public origin; the legacy WORKER_URL is accepted only for the bounded migration and logs a startup deprecation warning. PUBLIC_URL wins whenever both are set, and the installer drops WORKER_URL from freshly written env files when PUBLIC_URL is present.
• Startup and install validate the origin shape: absolute http/https URL with scheme and authority only; userinfo, query, fragment, and any path other than / are rejected.
• Plain-HTTP PUBLIC_URL on a non-loopback host is rejected in an explicitly production environment (DEPLOY_ENV, RSMON_ENV, or GO_ENV = production); other environments keep the historical warning.
• internal/wire adds public_url to WorkerInit (control plane to worker), keeping the legacy url field for old control planes; the worker prefers public_url and rejects unusable values, keeping the previous accepted URL. RegisterRequest.public_url is the registration contract for the pending RSMon counterpart (the worker does not currently transmit the URL during registration; it consumes the accepted endpoint from WorkerInit).
• The legacy WORKER_URL is held only to the tolerant absolute-URL check (no newly rejected legacy shapes); PUBLIC_URL is held to the strict scheme-and-authority origin shape. Both reject a missing hostname, e.g. https://:27401.

2026-07-19

Standalone installation and deployment

• Added rsmon-worker install for installing the current binary, a mode-0600 environment file, and a root-owned systemd service.
• Added rsmon-worker deploy for installing workers over SSH with key or password authentication, optional secret files, and host-key verification through known_hosts or a pinned fingerprint.
• Added --token-file, --url, --api-key, and --no-start deployment options.
• Added optional --docker deployment using a prebuilt image. Docker install and deploy now require an immutable repository@sha256:... reference before any Docker or remote-host mutation; the former mutable latest default is no longer accepted.
• Simplified the default systemd service to Type=simple, User=root, and Restart=on-failure.
• Reworked the legacy scripts/install-systemd.sh script as a compatibility wrapper around the built-in installer.

Standalone repository cleanup

• Removed the remaining certificate-bundle fallback under /data/rsmon and updated its documentation.
• Changed Docker build and runtime bases to public Go and Debian images.
• Documented binary, systemd, Docker, and SSH deployment workflows.
• Added installer and SSH host-verification tests.

Verification

• Passed make test, make build, go vet, and go mod verify.
• Validated systemd units, Compose configuration, Docker image pull and execution, and production worker job reporting.
• Published the changes as commit 3256dcd (feat: add worker install and deploy) on master, triggering the Docker image workflow.

Task protocol and local audit hardening

• Required one task-envelope branch, matching outer/inner job IDs, and a non-empty lease token before local execution.
• Added structured terminal failures for unsupported check kinds and safely attributable malformed envelopes.
• Recorded delegated notification outcomes in the bounded /notifications view using only job ID, method, status, duration, and time.
• Added static permanent handling for invalid deadlines and recovered notification executor panics without retaining secret-bearing text.
• Removed the critical-cluster test-config endpoint, CLI flag, environment switch, and production helper; hardcoded config application is test-only.
• Made runner token rotation connection-scoped and in-memory: it reconnects without stopping web, inventory, metrics, or cluster subsystems. Durable token storage, bootstrap exchange, rotation acknowledgement, and revocation remain unimplemented.
• Added bounded resend of dequeued check and notification result envelopes after websocket reconnect; control-plane application remains at-least-once and must deduplicate by leased job and lease token. Failed metric snapshots are dropped and replaced by the next periodic tick, not replayed.
• SIGTERM stops new dispatch and waits for active work, but stale-lease acknowledgement, bounded graceful final-result drain, and duplicate-frame coverage remain open.