worker/docs/source-installation.md

# SSH Source Installation Plan

## Status

In progress. Work packages 1-5 are implemented (work package 5 is the
Docker/OpenSSH E2E matrix wired into CI):

- `internal/installer/harness` builds and runs real OpenSSH containers
  for Alpine, Ubuntu, and Arch, waits for real network readiness, captures
  the server host key into a temp `known_hosts` file, and tears the
  container, network, per-instance fixture image tag, and temp dir down
  reliably. It never mocks SSH and connects with the
  `golang.org/x/crypto/ssh` library and known_hosts verification
  semantics the installer's `deploy` path relies on.
- `internal/sshinstall` resolves a remote host's distro, package manager,
  and init system from `/etc/os-release`, plans the pinned Go 1.26
  toolchain (published SHA-256) for the remote architecture, and produces
  a pure source-install plan. It executes nothing.
- `installer.SourceInstall` (work package 3) executes the plan through
  the same SSH transport, authentication, and host-key verification the
  `deploy` command uses. It installs the minimal package prerequisites,
  downloads and SHA-256-verifies the pinned Go toolchain before
  extraction, clones/updates the public repository, checks out the
  resolved branch, records the resolved branch and commit, and builds the
  worker to a staging path.
- Work package 4 (`installer.SourceInstall` with `Activation` enabled)
  atomically installs the staged binary, validated environment, data
  directory, and the detected init's service definition; starts/restarts
  the worker; verifies the process and `/healthz`; and rolls back to the
  prior working install on any activation/start/health failure. Reruns
  are idempotent with exactly one running worker and no leaked temp
  files. The CLI runs the full flow by default and accepts
  `--no-activate` (staging only) and `--no-start`.

The current Go installer can also upload a binary or deploy an immutable
Docker image over SSH (`deploy`). Source installs build remotely, then
activate the staged build atomically with rollback. Existing tests are
unit tests plus the harness tests that run against live OpenSSH
containers when explicitly enabled.

## Initial Platform Scope

The first source installer supports Linux and is validated on Alpine, Ubuntu,
and Arch Linux containers. CentOS-family support follows after its package and
service differences are implemented. Windows and macOS remain later platform
work despite the worker being written in Go.

Use `reg.rsxx.ru` image mirrors where available. Tests must not depend on Docker
Hub when a local mirror exists.

## Source Install Flow

The Go CLI connects through `golang.org/x/crypto/ssh` using existing key,
passphrase, password, sudo-password, known-hosts, and pinned-fingerprint support.
It then:

1. detects supported OS, architecture, package manager, init system, and
   privilege path;
2. installs only required packages (`git`, CA certificates, download/archive
   tools); it does not install `build-essential` or a C compiler unless a
   detected dependency requires CGO;
3. downloads the pinned Go 1.26 toolchain for the detected architecture and
   verifies the published SHA-256;
4. clones `https://rocketgit.ru/rsmon/worker.git` or updates an existing clone;
5. checks out the resolved branch (the pinned branch when one is configured,
   otherwise the remote's default branch) and records the resolved commit;
6. builds a reproducible worker binary with the repository build flags;
7. atomically installs the binary, validated environment, data directory, and
   service definition;
8. starts the service and verifies process status and `/healthz`.
Repository, branch, Go version, checksum source, build directory, and Go module
proxy may be configurable, but production output records their resolved values.
The default repository is publicly readable and requires no source credential.

## Work Package 3: Remote Execution To A Staging Path

Work package 3 is `installer.SourceInstall` in
`internal/installer/sourceinstall.go`. It reuses the `deploy` command's
`SSHOptions` (authentication, sudo password, known-hosts and fingerprint
verification) and runs every remote step with the same privilege path
(root, passwordless sudo, or `sudo -S -p ''` with the password delivered
only over stdin). Steps 1-6 of the flow above are implemented; step 7
(atomic install) and step 8 (start + verify) are work package 4.

Per step:

- **Prerequisite install.** `packageScript` renders the distro's
  idempotent command (`apk add --no-cache`, `apt-get update` +
  `apt-get install -y --no-install-recommends`, `pacman -Sy --noconfirm
  --needed`, `dnf install -y`) for the minimal plan packages (`git`,
  `ca-certificates`, `curl`, `tar`, `gzip`). No compiler is ever planned
  or installed.
- **Toolchain.** `toolchainScript` downloads the pinned Go tarball into a
  `mktemp` temp dir, verifies it with `sha256sum -c -` *before*
  extraction, extracts into a staging dir on the same filesystem as
  `ToolchainDir` (which must end in `/go`, default `/usr/local/go`),
  verifies the staged toolchain reports the target version, and only then
  swaps it into place. The prior toolchain is moved to a sibling
  `.go-backup` and is restored if the swap fails, so a failed
  download/verify/extract/swap always leaves the prior Go untouched. A
  present toolchain that already reports the target version is reused, so
  reruns do not re-download. Temp, staging, and backup directories are
  removed on success and failure.
- **Clone/update.** `cloneUpdateScript` clones the repository when
  `BuildDir` has no `.git` and otherwise fetches with `--prune`, so a
  rerun updates in place. An existing checkout's `remote.origin.url` must
  exactly match the configured repository before anything is fetched or
  built, so the installer can never fetch or build an unconfigured
  repository. The clone/fetch retries up to three times (2s apart)
  because real repositories can be transiently unreachable (DNS, TLS, or
  proxy hiccups); three bounded attempts keep a momentary outage from
  failing a full source install.
- **Resolved branch.** The installer resolves the remote default branch
  via `git remote set-head origin --auto` +
  `git symbolic-ref --short refs/remotes/origin/HEAD`. When no branch is
  pinned it builds the remote default (the public repo currently
  publishes `master`); a pinned branch must exist remotely or the install
  fails before the build. The resolved branch and the `git rev-parse
  HEAD` commit (validated as 40 lowercase hex) are returned by
  `SourceInstall`.
- **Staging build.** `buildScript` builds with `CGO_ENABLED=0`,
  `-trimpath`, the repository's own `-ldflags` shape (version `dev`,
  resolved commit short form, UTC build date), and both `GOCACHE` and
  `GOMODCACHE` inside the build dir (so reruns reuse them), plus an
  optional `GOPROXY`. The binary is built to a sibling `<stage>.new`,
  verified with `<stage>.new --version`, and only then atomically swapped
  over `<BuildDir>/rsmon-worker` (or `StageBinary`), so a failed build
  never replaces the previous staging binary. It is not written to
  `/usr/local/bin`.
- **Commit record.** The `rsmon-worker.commit` record (a
  `branch=...` / `commit=...` format in the build dir) is written only
  *after* a successful build, so the record and the staged binary always
  correspond to the same commit.

Security properties of work package 3:

- Every interpolated value (repository, branch, build dir, URLs, SHA-256,
  package names, paths) is single-quoted; repository, branch, commit, Go
  version, and Go architecture values are additionally validated with
  strict patterns. No worker token or control-plane credential is ever
  sent: the install stages a binary and touches no service configuration.
- The repository must be an `https://` URL without userinfo, so source
  credentials cannot reach the remote clone command or the clone's
  config.
- Sudo passwords are delivered over the session's stdin only, never in a
  command string (the same `sudoWrap` path the `deploy` command uses).
  Direct `--password`/`--sudo-password`/`--key-passphrase` flags remain
  available but expose the value through the process list and shell
  history; the CLI docs strongly prefer the `-file` variants. The source
  installer sends no worker token at all.
- Every step script fails closed: `set -eu` (or an explicit retry that
  exits non-zero) is used, so a failed checkout or fetch can never be
  masked by a stale subsequent command. The checkout step refuses before
  the destructive `checkout -B` when the tracked working tree is dirty
  (`git diff --quiet` / `--cached --quiet`), because `checkout -B` would
  silently discard local changes; a dirty-tree rerun fails at checkout and
  leaves the previous staging binary and commit record untouched.
- Remote errors are bounded: each step returns a step-labelled error,
  stderr and captured stdout are size-bounded in `runRemoteOutput`, and
  each remote command is capped by `--session-timeout` (default 30m).
- Toolchain temp, staging, and backup directories are removed on success
  and failure, and the acceptance test asserts no `/tmp/rsmon-toolchain-*`
  or `/usr/local/.go-staging-*`/`.go-backup` leaks after the rerun.
- Failed builds and failed checkouts leave the previous staging binary
  untouched (the binary is only overwritten by an atomically-swapped
  successful build, and a checkout failure aborts before the build).

## Work Package 4: Atomic Activation And Rollback

Work package 4 is the second half of `installer.SourceInstall`, driven by
`Activation` (`internal/installer/sourceactivate.go`). It runs over the
same SSH executor, privilege path, and bounded-error machinery as the
staging steps and installs the staged build atomically:

- **Layout.** The classic installer's on-disk layout is reused
  (`resolvePaths`): binary `/usr/local/bin/rsmon-worker[-name]`, config
  `/etc/rsmon-worker[-name]/worker.env` (mode 0600), data
  `/var/lib/rsmon-worker[-name]`, and a service definition for the
  detected init (systemd unit `/etc/systemd/system/rsmon-worker.service`
  mode 0644, or an OpenRC script `/etc/init.d/rsmon-worker` mode 0755).
  A named instance (`--name`) mirrors the classic multi-instance
  convention with `-<name>` suffixes and its own unit and port.
- **Environment.** The env file is rendered by the classic installer's
  `resolveInstallEnv` + `renderEnvFile` (so `PUBLIC_URL` canonicalization,
  basic-auth XOR, required `RSMON_URL`/`RSMON_TOKEN`, and systemd-safe
  value validation are identical to `install`). The env file is read
  *exactly once* and validated from the in-memory bytes
  (`parseEnvironmentContent`), and the rendered env is cached during
  option normalization and reused at activation time, so a hostile local
  writer cannot swap the file between the preflight and the remote
  install (env-file TOCTOU). The rendered bytes are uploaded into a
  server-created 0700 temp dir (`mktemp -d /tmp/rsmon-worker-act.XXXXXX`
  under the SSH user) rather than a predictable `/tmp` path, eliminating
  the symlink/TOCTOU attack surface on the upload while the token still
  travels only as base64 over the session stdin and, later, only inside
  the mode-0600 env file. The env is then installed with mode 0600 in the
  config dir (mode 0750).
- **Atomic install.** The staged binary is validated (`<stage>
  --version`) before any state is touched, then installed with a
  same-directory temp file + rename. The env and unit files are installed
  the same way. The data dir (and `webapp/` subdir) is created with mode
  0755.
- **Supervisor.** `restart_svc` drives start/restart through whichever
  init system is *actually running*: systemd (`systemctl`), OpenRC
  (`rc-service`), or the embedded supervisor fallback when no init is
  running (containers/chroots, the "no-service gate"). The health loop
  verifies the service is active *through the supervisor* — `systemctl
  is-active` for systemd, `rc-service status` for OpenRC, and a
  zombie-aware pid check for the embedded supervisor — never through a
  pid file on the init-managed paths. The embedded supervisor keeps the
  worker as a single background process with a pid file under the data
  dir, stops the previous instance before starting, refuses to kill a pid
  whose executable is not the configured worker (`/proc/<pid>/exe`
  checked before every `kill`), and answers `/healthz` via the worker's
  own `liveness` subcommand with a clean `env -i` environment built from
  the env file (so a rolled-back env can never leak a stale variable into
  the restored process). The OpenRC init script exports the env-file
  variables (`set -a` before sourcing) so the worker inherits them when
  OpenRC starts it.
- **Rollback.** The prior binary, env, and unit (and the unit's enable
  state) are snapshotted into the data dir before any mutation, and an
  `EXIT`/`HUP`/`INT`/`TERM` trap restores them (preserving the prior
  binary/env/unit metadata via `cp -p`) and restarts the prior service if
  any later step fails or a signal interrupts the run. A corrupt staged
  binary fails the pre-validation *before* the trap is armed, so the
  prior install is never touched. A fresh-install failure disables the
  newly-enabled unit (`systemctl disable` / `rc-update del`) and removes
  it; a rerun failure restores the prior unit and re-applies its prior
  enable state. Rollback also removes the backup dir, the `.new` temp
  files, the uploaded env/unit temps, and the activation lock.
- **Lock, recovery, interruption.** A `mkdir`-based activation lock
  (`.rsmon-activate.lock`, broken automatically when its recorded pid is
  dead) prevents concurrent activations from racing the deployed-state
  mutation. The snapshot is marked; a run killed mid-flight (SSH drop,
  SIGKILL) leaves that marker, and the next activation restores the
  leftover snapshot before proceeding, so the host never stays
  half-activated.
- **Idempotency.** A rerun reuses the existing env (rewriting it
  deterministically from the same knobs), swaps the binary atomically,
  restarts exactly one worker, and leaves no `.rsmon-backup`, `.new`,
  `/tmp/rsmon-worker-act.*`, or lock leftovers. `--no-start` installs the
  full layout without starting anything and reruns stay idle.

The `source-install` CLI activates by default; pass `--no-activate` for
the staging-only behavior of work package 3. The worker token is required
for activation and is supplied with `--token`/`--token-file` or
`--env-file` (prefer the file options; direct flags expose the value
through the process list).

## Work Package 5: Alpine/Ubuntu/Arch E2E In CI

Work package 5 runs the full Docker/OpenSSH E2E matrix on pushes to
`master` and on manual `workflow_dispatch` runs. It is a separate Gitea
Actions workflow (`.github/workflows/test-ssh.yml`) so the ordinary unit
CI run stays Docker-free; the job executes `make test-ssh` through
`scripts/ci/test-ssh.sh`.

CI behavior:

- **Trust boundary and triggers.** The workflow runs *only* on pushes to
  `master` (the default branch) and on manual `workflow_dispatch`. It
  deliberately does **not** trigger on `pull_request`: the distro fixtures
  execute the checked-out repository code inside privileged Docker
  containers, so an untrusted PR must never reach the runner's Docker
  surface automatically. Gitea's read-only token clamp for fork PRs does
  not limit what containers can do on the runner host, so PR coverage is
  left to the Docker-free unit CI (`ci.yml`) and to manual dispatch after
  a human reviews the change.
- **Concurrency is scoped per ref.** `concurrency.group:
  test-ssh-${{ gitea.ref }}` gives master pushes their own group (a newer
  master push cancels a superseded in-flight master run instead of
  stacking) and gives a manual dispatch on another branch its own group so
  it never cancels the master run. Gitea Actions evaluates the expression,
  and `gitea.ref` is the same documented context the repo's `docker.yml`
  already uses.
- **Least privilege.** The workflow declares `permissions: contents: read`
  (supported by Gitea Actions as the `GITEA_TOKEN` scope for
  code/releases), so the job's token can only read the repository; the
  workflow never writes, pushes, or publishes.
- **Action revisions.** The two actions this workflow uses are pinned to
  immutable full commit SHAs (not moving tags): `actions/checkout@v4` ->
  `11d5960a326750d5838078e36cf38b85af677262` and `actions/setup-go@v5` ->
  `40f1582b2485089dde7abd97c1529aa768e1baff` (verified 2026-08-13 against
  the GitHub API that each tag points to a commit object). The repo-wide
  convention still leaves `ci.yml` and `docker.yml` on moving tags
  (`@v4`, `@v5`, `@v3`, `@v6`); that is an accepted, documented risk: a
  tag move can change behavior without a workflow diff. New workflows
  should pin SHAs like this one; migrating the existing workflows is a
  separate change.
- **Runner requirement.** The harness dials fixture SSH ports published on
  the Docker daemon's `127.0.0.1`, so the job must share the daemon's
  loopback (a host-mode runner or a job container with host networking).
  `scripts/ci/test-ssh.sh` probes this with a tiny `docker run -p
  127.0.0.1::22` round trip *before* the matrix and fails fast with a
  clear, actionable message (including diagnostics and fix options)
  instead of after a 60m go-test timeout.
- **Bounding and timeout.** The workflow sets `timeout-minutes: 90` and
  the existing `make test-ssh` go-test `-timeout 60m` stays in place, so
  the whole job is hard-bounded even under slow networks or downloads.
- **Cleanup.** The harness already tears down every container, network,
  per-instance fixture image tag, and temp dir on success and failure.
  `scripts/ci/test-ssh.sh` additionally traps `EXIT` to remove any
  leftover `rsmon-worker-test-*` resource (container, network, or the
  per-instance `rsmon-worker-test/<fixture>-<suffix>:local` image tag), and
  the workflow adds an `if: always()` step that does the same even when the
  script itself is killed. All filters are anchored to the harness's own
  prefix and image repository (`name=^rsmon-worker-test-`,
  `reference=rsmon-worker-test/*`), so cleanup never touches a shared base
  image (`reg.rsxx.ru/library/alpine:3`, `ubuntu:24.04`, `archlinux:latest`)
  or an unrelated resource.
- **Mirrors and overrides.** The Alpine fixture already uses the
  `reg.rsxx.ru/library/alpine:3` mirror. Ubuntu and Arch have no mirror yet
  and default to Docker Hub. The workflow wires the
  `RSMON_TEST_IMAGE_ALPINE`, `RSMON_TEST_IMAGE_UBUNTU`, and
  `RSMON_TEST_IMAGE_ARCH` repository variables (empty by default) so an
  operator can pin a mirror or a specific distro snapshot per fixture, and
  the `RSMON_TEST_DOCKER_DNS` repository variable (comma-separated
  nameservers, applied as `docker run --dns ...`) to pin a resolver for
  flaky CI DNS. External network (go.dev toolchain download, the public
  rocketgit.ru source clone, distro package repos) is fetched live by
  design.
- **Artifacts and secrets.** The workflow uploads no artifacts: the go-test
  log stays in the runner's job log and nothing private (test key,
  `known_hosts`, env files) is retained on the runner or published. The
  fixture containers receive only the fixed `e2e-activation-test-token`
  and the bundled test-only key, never a real worker token.
- **Caching.** `actions/setup-go` caches the Go module/build cache used to
  compile the harness test binary (public dependencies only). Fixture and
  toolchain downloads are not cached because they run inside disposable
  distro containers; reruns rebuild them cleanly.

Work package 5 does not change how the source installer behaves. It only
adds a CI surface for the existing acceptance tests.

## Docker OpenSSH Test Harness

Adapt the real-network pattern from `/data/_swap/sshkeymanager`: start an
OpenSSH container, wait for SSH readiness, connect with the Go installer, and
tear the environment down reliably. Do not mock SSH command execution in the
acceptance test.

Provide images/fixtures for:

- Ubuntu with apt and systemd-compatible service testing where practical;
- Alpine with apk/OpenRC or a clearly separated no-service build/install gate;
- Arch with pacman and its service behavior.

Each clean target begins without Go or the worker source. The test asserts
package installation, verified Go version, clone branch/resolved commit, build,
atomic config permissions, running service where supported, and HTTP liveness.

### Implementation (`internal/installer/harness`)

The harness starts one container per distro fixture
(`testdata/fixtures/{alpine,ubuntu,arch}/Dockerfile`), waits for a real
TCP + SSH handshake, captures the server host key into a temp
`known_hosts` file, and connects with the `golang.org/x/crypto/ssh`
library and known_hosts verification semantics the installer's `deploy`
uses. The harness owns its connection code rather than calling into the
installer package, so the tests stay independent; only the library and
the verification semantics are shared.

Host-key handling is trust-on-first-use (TOFU): the fresh temp
`known_hosts` file accepts the first key the server presents. What the
harness proves is that a known_hosts entry carrying a *different* key is
rejected before any command runs (the `TestHarnessHostKeyMismatch` test),
not that a fingerprint is pinned.

The fixtures install over the public internet, so environments with flaky
local resolvers can pin a reliable upstream via the comma-separated
`RSMON_TEST_DOCKER_DNS` variable (applied as `docker run --dns ...`); it
is empty by default, keeping Docker's embedded DNS.

Teardown (`docker rm -f` + `docker network rm` + per-instance
`docker image rm` + temp-dir removal) is idempotent, runs on every
`Start` error path, and is verified by a dedicated test. Each harness
instance builds its own uniquely-tagged fixture image
(`rsmon-worker-test/<fixture>-<suffix>:local`), so removing one never
deletes a shared base image or another instance's image.

Base images, mirror-first:

| Fixture | Default image | Note |
| --- | --- | --- |
| alpine | `reg.rsxx.ru/library/alpine:3` | reg.rsxx.ru mirror exists |
| ubuntu | `ubuntu:24.04` | no mirror yet; override `RSMON_TEST_IMAGE_UBUNTU` |
| arch | `archlinux:latest` | no mirror yet; override `RSMON_TEST_IMAGE_ARCH` |

Any `RSMON_TEST_IMAGE_<NAME>` environment variable overrides the fixture
image, so a mirror or local cache can be used when available.

> **Mutable test images.** The fixture base tags above are deliberately
> mutable (a major-tag mirror ref and Docker Hub rolling tags) so the
> fixtures track current distro releases. Fixture builds are therefore
> not byte-reproducible; the worker's *source-install* production output
> pins immutable artifacts (a Go toolchain SHA-256, a branch's resolved
> commit) and this harness's image override is the escape hatch for
> reproducing a specific distro snapshot.

The fixtures authenticate with the bundled test key
(`testdata/keys/rsmon_test_ed25519`); password auth is disabled and
`PermitRootLogin` is `prohibit-password`. `openrc` is installed in the
Alpine fixture so init detection has a stable marker; Ubuntu and Arch
carry systemd markers.

> **Test-only key.** The bundled keypair is strictly a test fixture: it
> grants root SSH access only to the disposable containers that bake its
> public key. It must never be used for real hosts, added to production
> images, or treated as a credential outside the harness.

### Opt-in integration test controls

Ordinary unit runs never pull or start Docker. The Docker/OpenSSH tests
are gated behind the `RSMON_TEST_DOCKER` environment variable:

- `make test` (default CI unit run) pins `RSMON_TEST_DOCKER=0` and skips
  every harness Docker test, even when the flag is exported in the
  developer's environment.
- `make test-ssh` sets `RSMON_TEST_DOCKER=1` and runs the full fixture
  matrix (`TestHarnessFixtures` for alpine/ubuntu/arch, host-key
  mismatch, stable host key, failed-start cleanup, and teardown
  assertions).
- The harness `Start` itself refuses to run without the opt-in flag.

Run them locally with:

```bash
make test-ssh
```

or, equivalently:

```bash
RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harness
```

## Idempotency And Security

- A second run updates/fetches safely: the toolchain is reused when the version
  matches, the clone's origin is verified against the configured repository and
  then fetched in place, the resolved branch/commit record is rewritten after
  the new build succeeds, and the staging build atomically swaps over the
  previous staging binary. Activation (work package 4) makes the rerun
  idempotent at the service level too: exactly one worker process exists, the
  env file is rewritten deterministically from the same knobs, and no
  `.rsmon-backup`, `.new`, `/tmp/rsmon-worker-act.*`, or lock files leak.
- Wrong host fingerprints fail before remote mutation. The harness's fresh
  known_hosts file is trust-on-first-use; its dedicated mismatch test dials
  against a known_hosts entry carrying a different server key and proves the
  dial fails before any command runs.
- Tokens/passwords should come from files: `--key-passphrase-file`,
  `--password-file`, `--sudo-password-file`, and the activation
  `--token-file`/`--env-file` keep secrets out of argv and shell history,
  while the equivalent direct flags expose them through the process list. The
  worker token is written only to the mode-0600 env file, never echoed into a
  command or log, and sudo passwords travel only over the session's stdin. The
  env and unit uploads land in a server-created 0700 `mktemp -d` directory (not
  a predictable `/tmp` path), so no local user can plant a symlink at the
  upload target (TOCTOU), and the env file itself is read/rendered exactly once
  so a local writer cannot swap it between validation and upload.
- Remote temporary files are removed on success and failure, including the
  activation backup dir, the uploaded env/unit temps, and the activation lock.
  The snapshot is written under a marker so a run killed mid-flight is
  recovered (restored) by the next activation instead of leaving the host
  half-activated.
- Failed builds, failed checkouts, and failed toolchain swaps do not replace a
  working binary or service definition. Failed activations, failed starts, and
  failed `/healthz` verifications restore the prior binary, env, and service
  definition (preserving their metadata), re-apply the prior unit enable state
  (or disable a freshly-enabled unit on a fresh failure), and bring the prior
  worker back up (work package 4 rollback). The supervisor only ever kills a
  process whose `/proc/<pid>/exe` matches the configured worker binary, so a
  stale or recycled pid file can never kill an unrelated process.
- Package-manager and download failures return bounded actionable errors.
- The installer verifies Go tarball checksum before extraction.

## Implementation Work Packages

- [x] 1. Add reusable Docker/OpenSSH harness and distro fixtures.
- [x] 2. Add pure distro/toolchain/source-install script planning and unit tests
      (detection + planning foundation; remote execution is work package 3).
- [x] 3. Execute source installation through the existing SSH transport
      (prerequisites, verified Go toolchain, clone/update, resolved branch and
      commit record, and a build to a staging path; no service activation).
- [x] 4. Add atomic build/install, idempotency, and failure rollback (binary,
      env, data dir, and init service definition installed atomically; process
      and `/healthz` verified; activation/start/health failures restore the
      prior install; reruns keep exactly one service).
- [x] 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
- [ ] 6. Add CentOS-family support.
- [ ] 7. Plan native Windows service and macOS launchd installers separately.

## Acceptance Gates

- [x] All three initial Linux images install from a clean state through OpenSSH:
      package install, verified Go 1.26, clone/update, resolved commit, staging
      build, and (work package 4) atomic activation with a running verified
      worker.
- [x] The built worker reports the expected version/commit and serves `/healthz`
      (verified by the E2E fixture tests and by the activation health gate).
- [x] Re-running the installer succeeds without duplicate services or leaked
      files (exactly one worker process and no backup/temp leftovers asserted by
      the rerun test on all three fixtures).
- [x] Host-key, checksum, clone, build, and service-start failure tests preserve
      the previous installation (build/checkout, activation, start, and health
      failure reruns on the Alpine fixture all restore the prior binary, env,
      and running service).
- [x] CI uses approved registry mirrors and cleans every test container/network.

## Verified Test Evidence (work packages 1-4)

Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):

- Alpine `3.24.1` (mirror `reg.rsxx.ru/library/alpine:3`): distro alpine,
  pkg apk, init openrc.
- Ubuntu `24.04` (`VERSION_ID=24.04`): distro ubuntu, pkg apt, init systemd.
- Arch rolling image `archlinux:latest` (`VERSION_ID=20260809.0.570793`):
  distro arch, pkg pacman, init systemd.
- Host-key mismatch, host-key stability, failed-start cleanup, and complete
  teardown (container, network, fixture image tag, and temp dir gone) tests
  pass; no test container, network, or image tag is left behind.
- `TestSourceInstallFixtures` runs the full work-package-3 flow on each
  fixture from a clean state over the harness's real OpenSSH transport:
  prerequisite install, verified Go 1.26 toolchain download/extraction, clone
  of the public repository, resolution of its default branch, resolved-commit
  record file, and a staging build that reports the resolved commit via
  `--version`. A rerun succeeds, keeps the same branch, reuses the toolchain,
  and leaves no `/tmp/rsmon-toolchain-*` temp dirs.
- `TestSourceInstallActivationFixtures` runs the full work-package-4 flow on
  each fixture from a clean state: after the staging build the installer
  atomically installs the binary (`/usr/local/bin/rsmon-worker`), env
  (`/etc/rsmon-worker/worker.env`, mode 0600; config dir 0750), data dir
  (`/var/lib/rsmon-worker`), and the detected init's service definition
  (systemd unit on Ubuntu/Arch, OpenRC script on Alpine), then starts the
  worker via the embedded supervisor (no init runs inside the fixtures) and
  verifies exactly one worker process and a live `/healthz`. A rerun succeeds,
  keeps a single process, rewrites the env deterministically, and leaves no
  `.rsmon-backup`, `.new`, `/tmp/rsmon-worker-act.*`, or lock leftovers.
- `TestSourceInstallActivationFailureRollback` (Alpine) forces four rerun
  failures and proves each restores the prior install byte-for-byte and
  running: a dirty checkout (build failure), a sabotaged atomic swap
  (activation failure), `WORKER_CLUSTER_ENABLED=true` without credentials (the
  new process exits at boot; start failure), and `WORKER_HOST=255.255.255.255`
  (the new process runs but `/healthz` is unreachable; health failure). In
  every case the installed binary SHA-256, env, service definition, single
  process, and liveness match the pre-failure state, and no backup, lock, or
  temp files leak.
- `TestSourceInstallActivationNoStart` installs the full layout without
  starting anything, and reruns stay idle.
- The targeted shell fixture tests (`TestActivateShell*`) execute the real
  activation script against stub `systemctl`/`rc-service`/`rc-update`
  implementations and cover the init-managed paths the Docker fixtures cannot
  reach: systemd unit install/enable with a `systemctl is-active`-based health
  loop (and no pid file), OpenRC install/enable with an `rc-service`-based
  health loop, fresh-failure rollback disabling a newly-enabled unit, rerun
  rollback restoring the prior unit and its enable state, stale-lock breaking,
  and interrupted-run recovery from a leftover backup marker.
- The unit suite covers the per-step remote scripts, the activation script
  (backup + recovery marker, activation lock, atomic install, rollback trap,
  supervisor, pid/zombie handling, pid-belongs-to-binary checks, enable-state
  restoration, secrets absent), OpenRC unit rendering (including env
  export), option validation, branch/commit parsing, sudo wrapping (password
  never in the command), and an in-process real-SSH orchestration flow with
  failure paths for missing pinned branches, build failures, detection
  failures, and activation/start/health failures.
- **Residual limitation (stated honestly):** live systemd and OpenRC cannot
  run inside the Docker/OpenSSH harness (the container PID 1 is sshd), so the
  init-managed supervisor paths are verified by the stub-based shell fixture
  tests above rather than against a real init. The stub tools simulate unit
  state and command flow, not real systemd/OpenRC unit semantics; a real
  init-system smoke test on a booted host remains a follow-up.

Work package 5 (CI) evidence, recorded 2026-08-13:

- `.github/workflows/test-ssh.yml` runs `scripts/ci/test-ssh.sh` on pushes
  to `master` and on manual `workflow_dispatch` only (never on
  `pull_request`, because the fixtures execute the checked-out code inside
  privileged Docker). Concurrency is scoped per ref
  (`test-ssh-${{ gitea.ref }}`), the job token is `contents: read`, and
  the script's loopback port-publishing probe passes on a host Docker
  daemon while its `EXIT` cleanup leaves zero leftover containers,
  networks, or fixture image tags.
- A full local `make test-ssh` run (the exact command the CI job executes)
  passes the complete harness suite: `TestHarnessFixtures` (alpine/ubuntu/
  arch), host-key mismatch and stability, teardown, `TestSourceInstallFixtures`
  (work package 3 staging on all three distros), `TestSourceInstallActivationFixtures`
  (work package 4 activation on all three), activation failure rollback,
  `--no-start`, and the dirty-checkout preservation test. Pinned
  `RSMON_TEST_DOCKER_DNS=1.1.1.1` was used because the local network's
  default resolvers intermittently time out on `rocketgit.ru`.
- `TestSourceInstallDirtyCheckoutPreservesStaging` and the rollback test's
  build-failure step now dirty the tracked tree deterministically (appending
  a marker line to `Makefile`) instead of `git checkout master~1 -- Makefile`,
  which depended on the last two commits differing and silently stopped
  dirtying the tree once a commit did not touch that file.