worker/docs/source-installation.md

SSH Source Installation Plan

Status

In progress. Work packages 1-4 are implemented:

• internal/installer/harness builds and runs real OpenSSH containers for Alpine, Ubuntu, and Arch, waits for real network readiness, captures the server host key into a temp known_hosts file, and tears the container, network, per-instance fixture image tag, and temp dir down reliably. It never mocks SSH and connects with the golang.org/x/crypto/ssh library and known_hosts verification semantics the installer's deploy path relies on.
• internal/sshinstall resolves a remote host's distro, package manager, and init system from /etc/os-release, plans the pinned Go 1.26 toolchain (published SHA-256) for the remote architecture, and produces a pure source-install plan. It executes nothing.
• installer.SourceInstall (work package 3) executes the plan through the same SSH transport, authentication, and host-key verification the deploy command uses. It installs the minimal package prerequisites, downloads and SHA-256-verifies the pinned Go toolchain before extraction, clones/updates the public repository, checks out the resolved branch, records the resolved branch and commit, and builds the worker to a staging path.
• Work package 4 (installer.SourceInstall with Activation enabled) atomically installs the staged binary, validated environment, data directory, and the detected init's service definition; starts/restarts the worker; verifies the process and /healthz; and rolls back to the prior working install on any activation/start/health failure. Reruns are idempotent with exactly one running worker and no leaked temp files. The CLI runs the full flow by default and accepts --no-activate (staging only) and --no-start.

The current Go installer can also upload a binary or deploy an immutable Docker image over SSH (deploy). Source installs build remotely, then activate the staged build atomically with rollback. Existing tests are unit tests plus the harness tests that run against live OpenSSH containers when explicitly enabled.

Initial Platform Scope

The first source installer supports Linux and is validated on Alpine, Ubuntu, and Arch Linux containers. CentOS-family support follows after its package and service differences are implemented. Windows and macOS remain later platform work despite the worker being written in Go.

Use reg.rsxx.ru image mirrors where available. Tests must not depend on Docker Hub when a local mirror exists.

Source Install Flow

The Go CLI connects through golang.org/x/crypto/ssh using existing key, passphrase, password, sudo-password, known-hosts, and pinned-fingerprint support. It then:

1. detects supported OS, architecture, package manager, init system, and privilege path;
2. installs only required packages (git, CA certificates, download/archive tools); it does not install build-essential or a C compiler unless a detected dependency requires CGO;
3. downloads the pinned Go 1.26 toolchain for the detected architecture and verifies the published SHA-256;
4. clones https://rocketgit.ru/rsmon/worker.git or updates an existing clone;
5. checks out the resolved branch (the pinned branch when one is configured, otherwise the remote's default branch) and records the resolved commit;
6. builds a reproducible worker binary with the repository build flags;
7. atomically installs the binary, validated environment, data directory, and service definition;
8. starts the service and verifies process status and /healthz. Repository, branch, Go version, checksum source, build directory, and Go module proxy may be configurable, but production output records their resolved values. The default repository is publicly readable and requires no source credential.

Work Package 3: Remote Execution To A Staging Path

Work package 3 is installer.SourceInstall in internal/installer/sourceinstall.go. It reuses the deploy command's SSHOptions (authentication, sudo password, known-hosts and fingerprint verification) and runs every remote step with the same privilege path (root, passwordless sudo, or sudo -S -p '' with the password delivered only over stdin). Steps 1-6 of the flow above are implemented; step 7 (atomic install) and step 8 (start + verify) are work package 4.

Per step:

• Prerequisite install. packageScript renders the distro's idempotent command (apk add --no-cache, apt-get update + apt-get install -y --no-install-recommends, pacman -Sy --noconfirm --needed, dnf install -y) for the minimal plan packages (git, ca-certificates, curl, tar, gzip). No compiler is ever planned or installed.
• Toolchain. toolchainScript downloads the pinned Go tarball into a mktemp temp dir, verifies it with sha256sum -c - before extraction, extracts into a staging dir on the same filesystem as ToolchainDir (which must end in /go, default /usr/local/go), verifies the staged toolchain reports the target version, and only then swaps it into place. The prior toolchain is moved to a sibling .go-backup and is restored if the swap fails, so a failed download/verify/extract/swap always leaves the prior Go untouched. A present toolchain that already reports the target version is reused, so reruns do not re-download. Temp, staging, and backup directories are removed on success and failure.
• Clone/update. cloneUpdateScript clones the repository when BuildDir has no .git and otherwise fetches with --prune, so a rerun updates in place. An existing checkout's remote.origin.url must exactly match the configured repository before anything is fetched or built, so the installer can never fetch or build an unconfigured repository. The clone/fetch retries up to three times (2s apart) because real repositories can be transiently unreachable (DNS, TLS, or proxy hiccups); three bounded attempts keep a momentary outage from failing a full source install.
• Resolved branch. The installer resolves the remote default branch via git remote set-head origin --auto + git symbolic-ref --short refs/remotes/origin/HEAD. When no branch is pinned it builds the remote default (the public repo currently publishes master); a pinned branch must exist remotely or the install fails before the build. The resolved branch and the git rev-parse HEAD commit (validated as 40 lowercase hex) are returned by SourceInstall.
• Staging build. buildScript builds with CGO_ENABLED=0, -trimpath, the repository's own -ldflags shape (version dev, resolved commit short form, UTC build date), and both GOCACHE and GOMODCACHE inside the build dir (so reruns reuse them), plus an optional GOPROXY. The binary is built to a sibling <stage>.new, verified with <stage>.new --version, and only then atomically swapped over <BuildDir>/rsmon-worker (or StageBinary), so a failed build never replaces the previous staging binary. It is not written to /usr/local/bin.
• Commit record. The rsmon-worker.commit record (a branch=... / commit=... format in the build dir) is written only after a successful build, so the record and the staged binary always correspond to the same commit.

Security properties of work package 3:

• Every interpolated value (repository, branch, build dir, URLs, SHA-256, package names, paths) is single-quoted; repository, branch, commit, Go version, and Go architecture values are additionally validated with strict patterns. No worker token or control-plane credential is ever sent: the install stages a binary and touches no service configuration.
• The repository must be an https:// URL without userinfo, so source credentials cannot reach the remote clone command or the clone's config.
• Sudo passwords are delivered over the session's stdin only, never in a command string (the same sudoWrap path the deploy command uses). Direct --password/--sudo-password/--key-passphrase flags remain available but expose the value through the process list and shell history; the CLI docs strongly prefer the -file variants. The source installer sends no worker token at all.
• Every step script fails closed: set -eu (or an explicit retry that exits non-zero) is used, so a failed checkout or fetch can never be masked by a stale subsequent command. The checkout step refuses before the destructive checkout -B when the tracked working tree is dirty (git diff --quiet / --cached --quiet), because checkout -B would silently discard local changes; a dirty-tree rerun fails at checkout and leaves the previous staging binary and commit record untouched.
• Remote errors are bounded: each step returns a step-labelled error, stderr and captured stdout are size-bounded in runRemoteOutput, and each remote command is capped by --session-timeout (default 30m).
• Toolchain temp, staging, and backup directories are removed on success and failure, and the acceptance test asserts no /tmp/rsmon-toolchain-* or /usr/local/.go-staging-*/.go-backup leaks after the rerun.
• Failed builds and failed checkouts leave the previous staging binary untouched (the binary is only overwritten by an atomically-swapped successful build, and a checkout failure aborts before the build).

Work Package 4: Atomic Activation And Rollback

Work package 4 is the second half of installer.SourceInstall, driven by Activation (internal/installer/sourceactivate.go). It runs over the same SSH executor, privilege path, and bounded-error machinery as the staging steps and installs the staged build atomically:

• Layout. The classic installer's on-disk layout is reused (resolvePaths): binary /usr/local/bin/rsmon-worker[-name], config /etc/rsmon-worker[-name]/worker.env (mode 0600), data /var/lib/rsmon-worker[-name], and a service definition for the detected init (systemd unit /etc/systemd/system/rsmon-worker.service mode 0644, or an OpenRC script /etc/init.d/rsmon-worker mode 0755). A named instance (--name) mirrors the classic multi-instance convention with -<name> suffixes and its own unit and port.
• Environment. The env file is rendered by the classic installer's resolveInstallEnv + renderEnvFile (so PUBLIC_URL canonicalization, basic-auth XOR, required RSMON_URL/RSMON_TOKEN, and systemd-safe value validation are identical to install). The env file is read exactly once and validated from the in-memory bytes (parseEnvironmentContent), and the rendered env is cached during option normalization and reused at activation time, so a hostile local writer cannot swap the file between the preflight and the remote install (env-file TOCTOU). The rendered bytes are uploaded into a server-created 0700 temp dir (mktemp -d /tmp/rsmon-worker-act.XXXXXX under the SSH user) rather than a predictable /tmp path, eliminating the symlink/TOCTOU attack surface on the upload while the token still travels only as base64 over the session stdin and, later, only inside the mode-0600 env file. The env is then installed with mode 0600 in the config dir (mode 0750).
• Atomic install. The staged binary is validated (<stage> --version) before any state is touched, then installed with a same-directory temp file + rename. The env and unit files are installed the same way. The data dir (and webapp/ subdir) is created with mode 0755.
• Supervisor. restart_svc drives start/restart through whichever init system is actually running: systemd (systemctl), OpenRC (rc-service), or the embedded supervisor fallback when no init is running (containers/chroots, the "no-service gate"). The health loop verifies the service is active through the supervisor — systemctl is-active for systemd, rc-service status for OpenRC, and a zombie-aware pid check for the embedded supervisor — never through a pid file on the init-managed paths. The embedded supervisor keeps the worker as a single background process with a pid file under the data dir, stops the previous instance before starting, refuses to kill a pid whose executable is not the configured worker (/proc/<pid>/exe checked before every kill), and answers /healthz via the worker's own liveness subcommand with a clean env -i environment built from the env file (so a rolled-back env can never leak a stale variable into the restored process). The OpenRC init script exports the env-file variables (set -a before sourcing) so the worker inherits them when OpenRC starts it.
• Rollback. The prior binary, env, and unit (and the unit's enable state) are snapshotted into the data dir before any mutation, and an EXIT/HUP/INT/TERM trap restores them (preserving the prior binary/env/unit metadata via cp -p) and restarts the prior service if any later step fails or a signal interrupts the run. A corrupt staged binary fails the pre-validation before the trap is armed, so the prior install is never touched. A fresh-install failure disables the newly-enabled unit (systemctl disable / rc-update del) and removes it; a rerun failure restores the prior unit and re-applies its prior enable state. Rollback also removes the backup dir, the .new temp files, the uploaded env/unit temps, and the activation lock.
• Lock, recovery, interruption. A mkdir-based activation lock (.rsmon-activate.lock, broken automatically when its recorded pid is dead) prevents concurrent activations from racing the deployed-state mutation. The snapshot is marked; a run killed mid-flight (SSH drop, SIGKILL) leaves that marker, and the next activation restores the leftover snapshot before proceeding, so the host never stays half-activated.
• Idempotency. A rerun reuses the existing env (rewriting it deterministically from the same knobs), swaps the binary atomically, restarts exactly one worker, and leaves no .rsmon-backup, .new, /tmp/rsmon-worker-act.*, or lock leftovers. --no-start installs the full layout without starting anything and reruns stay idle.

The source-install CLI activates by default; pass --no-activate for the staging-only behavior of work package 3. The worker token is required for activation and is supplied with --token/--token-file or --env-file (prefer the file options; direct flags expose the value through the process list).

Docker OpenSSH Test Harness

Adapt the real-network pattern from /data/_swap/sshkeymanager: start an OpenSSH container, wait for SSH readiness, connect with the Go installer, and tear the environment down reliably. Do not mock SSH command execution in the acceptance test.

Provide images/fixtures for:

• Ubuntu with apt and systemd-compatible service testing where practical;
• Alpine with apk/OpenRC or a clearly separated no-service build/install gate;
• Arch with pacman and its service behavior.

Each clean target begins without Go or the worker source. The test asserts package installation, verified Go version, clone branch/resolved commit, build, atomic config permissions, running service where supported, and HTTP liveness.

Implementation (internal/installer/harness)

The harness starts one container per distro fixture (testdata/fixtures/{alpine,ubuntu,arch}/Dockerfile), waits for a real TCP + SSH handshake, captures the server host key into a temp known_hosts file, and connects with the golang.org/x/crypto/ssh library and known_hosts verification semantics the installer's deploy uses. The harness owns its connection code rather than calling into the installer package, so the tests stay independent; only the library and the verification semantics are shared.

Host-key handling is trust-on-first-use (TOFU): the fresh temp known_hosts file accepts the first key the server presents. What the harness proves is that a known_hosts entry carrying a different key is rejected before any command runs (the TestHarnessHostKeyMismatch test), not that a fingerprint is pinned.

The fixtures install over the public internet, so environments with flaky local resolvers can pin a reliable upstream via the comma-separated RSMON_TEST_DOCKER_DNS variable (applied as docker run --dns ...); it is empty by default, keeping Docker's embedded DNS.

Teardown (docker rm -f + docker network rm + per-instance docker image rm + temp-dir removal) is idempotent, runs on every Start error path, and is verified by a dedicated test. Each harness instance builds its own uniquely-tagged fixture image (rsmon-worker-test/<fixture>-<suffix>:local), so removing one never deletes a shared base image or another instance's image.

Base images, mirror-first:

Fixture	Default image	Note
alpine	reg.rsxx.ru/library/alpine:3	reg.rsxx.ru mirror exists
ubuntu	ubuntu:24.04	no mirror yet; override RSMON_TEST_IMAGE_UBUNTU
arch	archlinux:latest	no mirror yet; override RSMON_TEST_IMAGE_ARCH

Any RSMON_TEST_IMAGE_<NAME> environment variable overrides the fixture image, so a mirror or local cache can be used when available.

Mutable test images. The fixture base tags above are deliberately mutable (a major-tag mirror ref and Docker Hub rolling tags) so the fixtures track current distro releases. Fixture builds are therefore not byte-reproducible; the worker's source-install production output pins immutable artifacts (a Go toolchain SHA-256, a branch's resolved commit) and this harness's image override is the escape hatch for reproducing a specific distro snapshot.

The fixtures authenticate with the bundled test key (testdata/keys/rsmon_test_ed25519); password auth is disabled and PermitRootLogin is prohibit-password. openrc is installed in the Alpine fixture so init detection has a stable marker; Ubuntu and Arch carry systemd markers.

Test-only key. The bundled keypair is strictly a test fixture: it grants root SSH access only to the disposable containers that bake its public key. It must never be used for real hosts, added to production images, or treated as a credential outside the harness.

Opt-in integration test controls

Ordinary unit runs never pull or start Docker. The Docker/OpenSSH tests are gated behind the RSMON_TEST_DOCKER environment variable:

• make test (default CI unit run) pins RSMON_TEST_DOCKER=0 and skips every harness Docker test, even when the flag is exported in the developer's environment.
• make test-ssh sets RSMON_TEST_DOCKER=1 and runs the full fixture matrix (TestHarnessFixtures for alpine/ubuntu/arch, host-key mismatch, stable host key, failed-start cleanup, and teardown assertions).
• The harness Start itself refuses to run without the opt-in flag.

Run them locally with:

make test-ssh

or, equivalently:

RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harness

Idempotency And Security

• A second run updates/fetches safely: the toolchain is reused when the version matches, the clone's origin is verified against the configured repository and then fetched in place, the resolved branch/commit record is rewritten after the new build succeeds, and the staging build atomically swaps over the previous staging binary. Activation (work package 4) makes the rerun idempotent at the service level too: exactly one worker process exists, the env file is rewritten deterministically from the same knobs, and no .rsmon-backup, .new, /tmp/rsmon-worker-act.*, or lock files leak.
• Wrong host fingerprints fail before remote mutation. The harness's fresh known_hosts file is trust-on-first-use; its dedicated mismatch test dials against a known_hosts entry carrying a different server key and proves the dial fails before any command runs.
• Tokens/passwords should come from files: --key-passphrase-file, --password-file, --sudo-password-file, and the activation --token-file/--env-file keep secrets out of argv and shell history, while the equivalent direct flags expose them through the process list. The worker token is written only to the mode-0600 env file, never echoed into a command or log, and sudo passwords travel only over the session's stdin. The env and unit uploads land in a server-created 0700 mktemp -d directory (not a predictable /tmp path), so no local user can plant a symlink at the upload target (TOCTOU), and the env file itself is read/rendered exactly once so a local writer cannot swap it between validation and upload.
• Remote temporary files are removed on success and failure, including the activation backup dir, the uploaded env/unit temps, and the activation lock. The snapshot is written under a marker so a run killed mid-flight is recovered (restored) by the next activation instead of leaving the host half-activated.
• Failed builds, failed checkouts, and failed toolchain swaps do not replace a working binary or service definition. Failed activations, failed starts, and failed /healthz verifications restore the prior binary, env, and service definition (preserving their metadata), re-apply the prior unit enable state (or disable a freshly-enabled unit on a fresh failure), and bring the prior worker back up (work package 4 rollback). The supervisor only ever kills a process whose /proc/<pid>/exe matches the configured worker binary, so a stale or recycled pid file can never kill an unrelated process.
• Package-manager and download failures return bounded actionable errors.
• The installer verifies Go tarball checksum before extraction.

Implementation Work Packages

• 1. Add reusable Docker/OpenSSH harness and distro fixtures.
• 2. Add pure distro/toolchain/source-install script planning and unit tests (detection + planning foundation; remote execution is work package 3).
• 3. Execute source installation through the existing SSH transport (prerequisites, verified Go toolchain, clone/update, resolved branch and commit record, and a build to a staging path; no service activation).
• 4. Add atomic build/install, idempotency, and failure rollback (binary, env, data dir, and init service definition installed atomically; process and /healthz verified; activation/start/health failures restore the prior install; reruns keep exactly one service).
• 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
• 6. Add CentOS-family support.
• 7. Plan native Windows service and macOS launchd installers separately.

Acceptance Gates

• All three initial Linux images install from a clean state through OpenSSH: package install, verified Go 1.26, clone/update, resolved commit, staging build, and (work package 4) atomic activation with a running verified worker.
• The built worker reports the expected version/commit and serves /healthz (verified by the E2E fixture tests and by the activation health gate).
• Re-running the installer succeeds without duplicate services or leaked files (exactly one worker process and no backup/temp leftovers asserted by the rerun test on all three fixtures).
• Host-key, checksum, clone, build, and service-start failure tests preserve the previous installation (build/checkout, activation, start, and health failure reruns on the Alpine fixture all restore the prior binary, env, and running service).
• CI uses approved registry mirrors and cleans every test container/network.

Verified Test Evidence (work packages 1-4)

Recorded 2026-08-12 from make test-ssh (Docker Engine 29.7.1):

• Alpine 3.24.1 (mirror reg.rsxx.ru/library/alpine:3): distro alpine, pkg apk, init openrc.
• Ubuntu 24.04 (VERSION_ID=24.04): distro ubuntu, pkg apt, init systemd.
• Arch rolling image archlinux:latest (VERSION_ID=20260809.0.570793): distro arch, pkg pacman, init systemd.
• Host-key mismatch, host-key stability, failed-start cleanup, and complete teardown (container, network, fixture image tag, and temp dir gone) tests pass; no test container, network, or image tag is left behind.
• TestSourceInstallFixtures runs the full work-package-3 flow on each fixture from a clean state over the harness's real OpenSSH transport: prerequisite install, verified Go 1.26 toolchain download/extraction, clone of the public repository, resolution of its default branch, resolved-commit record file, and a staging build that reports the resolved commit via --version. A rerun succeeds, keeps the same branch, reuses the toolchain, and leaves no /tmp/rsmon-toolchain-* temp dirs.
• TestSourceInstallActivationFixtures runs the full work-package-4 flow on each fixture from a clean state: after the staging build the installer atomically installs the binary (/usr/local/bin/rsmon-worker), env (/etc/rsmon-worker/worker.env, mode 0600; config dir 0750), data dir (/var/lib/rsmon-worker), and the detected init's service definition (systemd unit on Ubuntu/Arch, OpenRC script on Alpine), then starts the worker via the embedded supervisor (no init runs inside the fixtures) and verifies exactly one worker process and a live /healthz. A rerun succeeds, keeps a single process, rewrites the env deterministically, and leaves no .rsmon-backup, .new, /tmp/rsmon-worker-act.*, or lock leftovers.
• TestSourceInstallActivationFailureRollback (Alpine) forces four rerun failures and proves each restores the prior install byte-for-byte and running: a dirty checkout (build failure), a sabotaged atomic swap (activation failure), WORKER_CLUSTER_ENABLED=true without credentials (the new process exits at boot; start failure), and WORKER_HOST=255.255.255.255 (the new process runs but /healthz is unreachable; health failure). In every case the installed binary SHA-256, env, service definition, single process, and liveness match the pre-failure state, and no backup, lock, or temp files leak.
• TestSourceInstallActivationNoStart installs the full layout without starting anything, and reruns stay idle.
• The targeted shell fixture tests (TestActivateShell*) execute the real activation script against stub systemctl/rc-service/rc-update implementations and cover the init-managed paths the Docker fixtures cannot reach: systemd unit install/enable with a systemctl is-active-based health loop (and no pid file), OpenRC install/enable with an rc-service-based health loop, fresh-failure rollback disabling a newly-enabled unit, rerun rollback restoring the prior unit and its enable state, stale-lock breaking, and interrupted-run recovery from a leftover backup marker.
• The unit suite covers the per-step remote scripts, the activation script (backup + recovery marker, activation lock, atomic install, rollback trap, supervisor, pid/zombie handling, pid-belongs-to-binary checks, enable-state restoration, secrets absent), OpenRC unit rendering (including env export), option validation, branch/commit parsing, sudo wrapping (password never in the command), and an in-process real-SSH orchestration flow with failure paths for missing pinned branches, build failures, detection failures, and activation/start/health failures.
• Residual limitation (stated honestly): live systemd and OpenRC cannot run inside the Docker/OpenSSH harness (the container PID 1 is sshd), so the init-managed supervisor paths are verified by the stub-based shell fixture tests above rather than against a real init. The stub tools simulate unit state and command flow, not real systemd/OpenRC unit semantics; a real init-system smoke test on a booted host remains a follow-up.