test(installer): add OpenSSH distro harness

docs/source-installation.md

@@ -2,10 +2,28 @@
 
 ## Status
 
-Planned. The current Go installer can upload a binary or deploy an immutable
-Docker image over SSH. It does not yet install build prerequisites, download Go,
-clone the public repository, or build remotely. Existing tests are unit tests;
-there is no live OpenSSH-container installation test.
+In progress. Work package 1 (reusable Docker/OpenSSH harness and distro
+fixtures) and the pure detection/planning foundation (work package 2
+core) are implemented:
+
+- `internal/installer/harness` builds and runs real OpenSSH containers
+  for Alpine, Ubuntu, and Arch, waits for real network readiness, captures
+  the server host key into a temp `known_hosts` file, and tears the
+  container, network, per-instance fixture image tag, and temp dir down
+  reliably. It never mocks SSH and connects with the
+  `golang.org/x/crypto/ssh` library and known_hosts verification
+  semantics the installer's `deploy` path relies on.
+- `internal/sshinstall` resolves a remote host's distro, package manager,
+  and init system from `/etc/os-release`, plans the pinned Go 1.26
+  toolchain (published SHA-256) for the remote architecture, and produces
+  a pure source-install plan. It executes nothing.
+
+The current Go installer can still only upload a binary or deploy an
+immutable Docker image over SSH. Remote package/toolchain/source build
+execution (work packages 3 and 4) is not implemented yet; the acceptance
+test currently stops after detection, clean-state, and planning
+assertions. Existing tests are unit tests; the harness tests run against
+live OpenSSH containers when explicitly enabled.
 
 ## Initial Platform Scope
 
@@ -58,10 +76,93 @@ Each clean target begins without Go or the worker source. The test asserts
 package installation, verified Go version, clone branch/resolved commit, build,
 atomic config permissions, running service where supported, and HTTP liveness.
 
+### Implementation (`internal/installer/harness`)
+
+The harness starts one container per distro fixture
+(`testdata/fixtures/{alpine,ubuntu,arch}/Dockerfile`), waits for a real
+TCP + SSH handshake, captures the server host key into a temp
+`known_hosts` file, and connects with the `golang.org/x/crypto/ssh`
+library and known_hosts verification semantics the installer's `deploy`
+uses. The harness owns its connection code rather than calling into the
+installer package, so the tests stay independent; only the library and
+the verification semantics are shared.
+
+Host-key handling is trust-on-first-use (TOFU): the fresh temp
+`known_hosts` file accepts the first key the server presents. What the
+harness proves is that a known_hosts entry carrying a *different* key is
+rejected before any command runs (the `TestHarnessHostKeyMismatch` test),
+not that a fingerprint is pinned.
+
+Teardown (`docker rm -f` + `docker network rm` + per-instance
+`docker image rm` + temp-dir removal) is idempotent, runs on every
+`Start` error path, and is verified by a dedicated test. Each harness
+instance builds its own uniquely-tagged fixture image
+(`rsmon-worker-test/<fixture>-<suffix>:local`), so removing one never
+deletes a shared base image or another instance's image.
+
+Base images, mirror-first:
+
+| Fixture | Default image | Note |
+| --- | --- | --- |
+| alpine | `reg.rsxx.ru/library/alpine:3` | reg.rsxx.ru mirror exists |
+| ubuntu | `ubuntu:24.04` | no mirror yet; override `RSMON_TEST_IMAGE_UBUNTU` |
+| arch | `archlinux:latest` | no mirror yet; override `RSMON_TEST_IMAGE_ARCH` |
+
+Any `RSMON_TEST_IMAGE_<NAME>` environment variable overrides the fixture
+image, so a mirror or local cache can be used when available.
+
+> **Mutable test images.** The fixture base tags above are deliberately
+> mutable (a major-tag mirror ref and Docker Hub rolling tags) so the
+> fixtures track current distro releases. Fixture builds are therefore
+> not byte-reproducible; the worker's *source-install* production output
+> pins immutable artifacts (a Go toolchain SHA-256, a branch's resolved
+> commit) and this harness's image override is the escape hatch for
+> reproducing a specific distro snapshot.
+
+The fixtures authenticate with the bundled test key
+(`testdata/keys/rsmon_test_ed25519`); password auth is disabled and
+`PermitRootLogin` is `prohibit-password`. `openrc` is installed in the
+Alpine fixture so init detection has a stable marker; Ubuntu and Arch
+carry systemd markers.
+
+> **Test-only key.** The bundled keypair is strictly a test fixture: it
+> grants root SSH access only to the disposable containers that bake its
+> public key. It must never be used for real hosts, added to production
+> images, or treated as a credential outside the harness.
+
+### Opt-in integration test controls
+
+Ordinary unit runs never pull or start Docker. The Docker/OpenSSH tests
+are gated behind the `RSMON_TEST_DOCKER` environment variable:
+
+- `make test` (default CI unit run) pins `RSMON_TEST_DOCKER=0` and skips
+  every harness Docker test, even when the flag is exported in the
+  developer's environment.
+- `make test-ssh` sets `RSMON_TEST_DOCKER=1` and runs the full fixture
+  matrix (`TestHarnessFixtures` for alpine/ubuntu/arch, host-key
+  mismatch, stable host key, failed-start cleanup, and teardown
+  assertions).
+- The harness `Start` itself refuses to run without the opt-in flag.
+
+Run them locally with:
+
+```bash
+make test-ssh
+```
+
+or, equivalently:
+
+```bash
+RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harness
+```
+
 ## Idempotency And Security
 
 - A second run updates/fetches safely and leaves one active service.
-- Wrong host fingerprints fail before remote mutation.
+- Wrong host fingerprints fail before remote mutation. The harness's fresh
+  known_hosts file is trust-on-first-use; its dedicated mismatch test dials
+  against a known_hosts entry carrying a different server key and proves the
+  dial fails before any command runs.
 - Tokens/passwords come from files or stdin-safe channels and never appear in
   command arguments, logs, source checkout, or shell history.
 - Remote temporary files are removed on success and failure.
@@ -71,19 +172,33 @@ atomic config permissions, running service where supported, and HTTP liveness.
 
 ## Implementation Work Packages
 
-1. Add reusable Docker/OpenSSH harness and distro fixtures.
-2. Add pure distro/toolchain/source-install script planning and unit tests.
-3. Execute source installation through the existing SSH transport.
-4. Add atomic build/install, idempotency, and failure rollback.
-5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
-6. Add CentOS-family support.
-7. Plan native Windows service and macOS launchd installers separately.
+- [x] 1. Add reusable Docker/OpenSSH harness and distro fixtures.
+- [x] 2. Add pure distro/toolchain/source-install script planning and unit tests
+      (detection + planning foundation; remote execution is work package 3).
+- [ ] 3. Execute source installation through the existing SSH transport.
+- [ ] 4. Add atomic build/install, idempotency, and failure rollback.
+- [ ] 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
+- [ ] 6. Add CentOS-family support.
+- [ ] 7. Plan native Windows service and macOS launchd installers separately.
 
 ## Acceptance Gates
 
-- All three initial Linux images install from a clean state through OpenSSH.
-- The built worker reports the expected version/commit and serves `/healthz`.
-- Re-running the installer succeeds without duplicate services or leaked files.
-- Host-key, checksum, clone, build, and service-start failure tests preserve the
-  previous installation.
-- CI uses approved registry mirrors and cleans every test container/network.
+- [ ] All three initial Linux images install from a clean state through OpenSSH.
+- [ ] The built worker reports the expected version/commit and serves `/healthz`.
+- [ ] Re-running the installer succeeds without duplicate services or leaked files.
+- [ ] Host-key, checksum, clone, build, and service-start failure tests preserve the
+      previous installation.
+- [ ] CI uses approved registry mirrors and cleans every test container/network.
+
+## Verified Test Evidence (work packages 1 and 2)
+
+Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
+
+- Alpine `3.24.1` (mirror `reg.rsxx.ru/library/alpine:3`): distro alpine,
+  pkg apk, init openrc.
+- Ubuntu `24.04` (`VERSION_ID=24.04`): distro ubuntu, pkg apt, init systemd.
+- Arch rolling image `archlinux:latest` (`VERSION_ID=20260809.0.570793`):
+  distro arch, pkg pacman, init systemd.
+- Host-key mismatch, host-key stability, failed-start cleanup, and complete
+  teardown (container, network, fixture image tag, and temp dir gone) tests
+  pass; no test container, network, or image tag is left behind.