test(installer): add OpenSSH distro harness

docs/README.md

@@ -28,7 +28,7 @@ disagree, update both in the same change or mark the discrepancy explicitly.
 | [network-diagnostics.md](network-diagnostics.md) | Confirmation checks and dedicated diagnostic tasks | Partial |
 | [critical-check-cluster.md](critical-check-cluster.md) | Raft-backed dispatchless critical checks | Scaffold only |
 | [public-endpoint-and-identity.md](public-endpoint-and-identity.md) | One HTTPS origin, peer status, CA/mTLS, managed Raft topology | Planned; peer HTTPS partial |
-| [source-installation.md](source-installation.md) | Go SSH source installer and Docker/OpenSSH test matrix | Planned |
+| [source-installation.md](source-installation.md) | Go SSH source installer and Docker/OpenSSH test matrix | Partial: harness + planning landed, remote build pending |
 | [implementation-roadmap.md](implementation-roadmap.md) | Ordered repository work packages and release gates | Active |
 | [source-plan-migration.md](source-plan-migration.md) | Source-to-target conversion ledger and resolved conflicts | Complete mapping |
 

docs/changelog.md

@@ -2,6 +2,42 @@
 
 ## 2026-08-12
 
+### Source-install harness and planning foundations (work packages 1-2)
+
+- Added `internal/installer/harness`: a reusable Docker/OpenSSH test harness
+  that builds real OpenSSH containers for Alpine, Ubuntu, and Arch, waits for
+  real network SSH readiness, captures the server host key into a temp
+  `known_hosts` file, and tears the container, network, per-instance fixture
+  image tag, and temp dir down reliably. It uses the `golang.org/x/crypto/ssh`
+  library and known_hosts verification semantics the installer's `deploy` path
+  relies on (the harness owns its connection code rather than reusing the
+  installer functions) and never mocks SSH. A fresh known_hosts file trusts
+  the first key (TOFU); the host-key mismatch test proves a different key is
+  rejected before any command runs.
+- Added distro fixtures under `internal/installer/harness/testdata/fixtures`.
+  Alpine defaults to the `reg.rsxx.ru/library/alpine:3` mirror; Ubuntu and Arch
+  fall back to Docker Hub refs overridable via `RSMON_TEST_IMAGE_<NAME>`.
+  Each fixture starts clean (no Go, no worker source) and authenticates with a
+  bundled test key; password auth is disabled. The test key is strictly
+  test-only - it grants root only to the disposable fixture containers - and
+  must never be used outside the harness.
+- Added opt-in integration controls: the Docker tests run only with
+  `RSMON_TEST_DOCKER=1` (`make test-ssh`); default `make test` and `go test
+  ./...` skip them and never pull or start containers. `make test` also pins
+  `RSMON_TEST_DOCKER=0` so an exported opt-in flag cannot leak into the unit
+  run.
+- Integration tests assert real SSH round trips, clean target state, distro /
+  package-manager / init detection per fixture, a full source plan including
+  the pinned Go toolchain, host-key mismatch rejection, host-key stability,
+  failed-start cleanup, and complete teardown (container, network, fixture
+  image tag, and temp dir gone).
+- Added `internal/sshinstall`: pure, unit-tested detection and planning for the
+  source installer - os-release parsing, distro/package-manager/init
+  resolution, `uname -m` to Go archive mapping, pinned Go 1.26 toolchain with
+  published SHA-256, and a reviewable ordered plan. No remote execution yet.
+- `make test` now includes the new packages; `make test-ssh` runs the live
+  fixture matrix.
+
 ### Public endpoint configuration (milestone 1 of public-endpoint-and-identity)
 
 - `PUBLIC_URL` is now the canonical advertised public origin; the legacy

docs/implementation-roadmap.md

@@ -19,6 +19,17 @@ Worker repository:
   `main`;
 - document immutable SHA and release tags as production defaults.
 
+Source-install foundations landed:
+
+- [x] reusable Docker/OpenSSH harness and distro fixtures for Alpine, Ubuntu,
+      and Arch (`internal/installer/harness`) with real SSH readiness and
+      reliable teardown, gated behind `RSMON_TEST_DOCKER` (`make test-ssh`);
+- [x] pure distro/toolchain/source-install planning (`internal/sshinstall`):
+      os-release detection, package-manager/init resolution, pinned Go 1.26
+      toolchain with published SHA-256, and a plan the executor will run;
+- [ ] remote package install, Go download, clone, and build execution through
+      the SSH transport (source-install work package 3);
+
 Gate: a push publishes `sha-<12>` and `latest` manifests for both platforms,
 and a container remains healthy when the control plane is unavailable.
 

docs/source-installation.md

@@ -2,10 +2,28 @@
 
 ## Status
 
-Planned. The current Go installer can upload a binary or deploy an immutable
-Docker image over SSH. It does not yet install build prerequisites, download Go,
-clone the public repository, or build remotely. Existing tests are unit tests;
-there is no live OpenSSH-container installation test.
+In progress. Work package 1 (reusable Docker/OpenSSH harness and distro
+fixtures) and the pure detection/planning foundation (work package 2
+core) are implemented:
+
+- `internal/installer/harness` builds and runs real OpenSSH containers
+  for Alpine, Ubuntu, and Arch, waits for real network readiness, captures
+  the server host key into a temp `known_hosts` file, and tears the
+  container, network, per-instance fixture image tag, and temp dir down
+  reliably. It never mocks SSH and connects with the
+  `golang.org/x/crypto/ssh` library and known_hosts verification
+  semantics the installer's `deploy` path relies on.
+- `internal/sshinstall` resolves a remote host's distro, package manager,
+  and init system from `/etc/os-release`, plans the pinned Go 1.26
+  toolchain (published SHA-256) for the remote architecture, and produces
+  a pure source-install plan. It executes nothing.
+
+The current Go installer can still only upload a binary or deploy an
+immutable Docker image over SSH. Remote package/toolchain/source build
+execution (work packages 3 and 4) is not implemented yet; the acceptance
+test currently stops after detection, clean-state, and planning
+assertions. Existing tests are unit tests; the harness tests run against
+live OpenSSH containers when explicitly enabled.
 
 ## Initial Platform Scope
 
@@ -58,10 +76,93 @@ Each clean target begins without Go or the worker source. The test asserts
 package installation, verified Go version, clone branch/resolved commit, build,
 atomic config permissions, running service where supported, and HTTP liveness.
 
+### Implementation (`internal/installer/harness`)
+
+The harness starts one container per distro fixture
+(`testdata/fixtures/{alpine,ubuntu,arch}/Dockerfile`), waits for a real
+TCP + SSH handshake, captures the server host key into a temp
+`known_hosts` file, and connects with the `golang.org/x/crypto/ssh`
+library and known_hosts verification semantics the installer's `deploy`
+uses. The harness owns its connection code rather than calling into the
+installer package, so the tests stay independent; only the library and
+the verification semantics are shared.
+
+Host-key handling is trust-on-first-use (TOFU): the fresh temp
+`known_hosts` file accepts the first key the server presents. What the
+harness proves is that a known_hosts entry carrying a *different* key is
+rejected before any command runs (the `TestHarnessHostKeyMismatch` test),
+not that a fingerprint is pinned.
+
+Teardown (`docker rm -f` + `docker network rm` + per-instance
+`docker image rm` + temp-dir removal) is idempotent, runs on every
+`Start` error path, and is verified by a dedicated test. Each harness
+instance builds its own uniquely-tagged fixture image
+(`rsmon-worker-test/<fixture>-<suffix>:local`), so removing one never
+deletes a shared base image or another instance's image.
+
+Base images, mirror-first:
+
+| Fixture | Default image | Note |
+| --- | --- | --- |
+| alpine | `reg.rsxx.ru/library/alpine:3` | reg.rsxx.ru mirror exists |
+| ubuntu | `ubuntu:24.04` | no mirror yet; override `RSMON_TEST_IMAGE_UBUNTU` |
+| arch | `archlinux:latest` | no mirror yet; override `RSMON_TEST_IMAGE_ARCH` |
+
+Any `RSMON_TEST_IMAGE_<NAME>` environment variable overrides the fixture
+image, so a mirror or local cache can be used when available.
+
+> **Mutable test images.** The fixture base tags above are deliberately
+> mutable (a major-tag mirror ref and Docker Hub rolling tags) so the
+> fixtures track current distro releases. Fixture builds are therefore
+> not byte-reproducible; the worker's *source-install* production output
+> pins immutable artifacts (a Go toolchain SHA-256, a branch's resolved
+> commit) and this harness's image override is the escape hatch for
+> reproducing a specific distro snapshot.
+
+The fixtures authenticate with the bundled test key
+(`testdata/keys/rsmon_test_ed25519`); password auth is disabled and
+`PermitRootLogin` is `prohibit-password`. `openrc` is installed in the
+Alpine fixture so init detection has a stable marker; Ubuntu and Arch
+carry systemd markers.
+
+> **Test-only key.** The bundled keypair is strictly a test fixture: it
+> grants root SSH access only to the disposable containers that bake its
+> public key. It must never be used for real hosts, added to production
+> images, or treated as a credential outside the harness.
+
+### Opt-in integration test controls
+
+Ordinary unit runs never pull or start Docker. The Docker/OpenSSH tests
+are gated behind the `RSMON_TEST_DOCKER` environment variable:
+
+- `make test` (default CI unit run) pins `RSMON_TEST_DOCKER=0` and skips
+  every harness Docker test, even when the flag is exported in the
+  developer's environment.
+- `make test-ssh` sets `RSMON_TEST_DOCKER=1` and runs the full fixture
+  matrix (`TestHarnessFixtures` for alpine/ubuntu/arch, host-key
+  mismatch, stable host key, failed-start cleanup, and teardown
+  assertions).
+- The harness `Start` itself refuses to run without the opt-in flag.
+
+Run them locally with:
+
+```bash
+make test-ssh
+```
+
+or, equivalently:
+
+```bash
+RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harness
+```
+
 ## Idempotency And Security
 
 - A second run updates/fetches safely and leaves one active service.
-- Wrong host fingerprints fail before remote mutation.
+- Wrong host fingerprints fail before remote mutation. The harness's fresh
+  known_hosts file is trust-on-first-use; its dedicated mismatch test dials
+  against a known_hosts entry carrying a different server key and proves the
+  dial fails before any command runs.
 - Tokens/passwords come from files or stdin-safe channels and never appear in
   command arguments, logs, source checkout, or shell history.
 - Remote temporary files are removed on success and failure.
@@ -71,19 +172,33 @@ atomic config permissions, running service where supported, and HTTP liveness.
 
 ## Implementation Work Packages
 
-1. Add reusable Docker/OpenSSH harness and distro fixtures.
-2. Add pure distro/toolchain/source-install script planning and unit tests.
-3. Execute source installation through the existing SSH transport.
-4. Add atomic build/install, idempotency, and failure rollback.
-5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
-6. Add CentOS-family support.
-7. Plan native Windows service and macOS launchd installers separately.
+- [x] 1. Add reusable Docker/OpenSSH harness and distro fixtures.
+- [x] 2. Add pure distro/toolchain/source-install script planning and unit tests
+      (detection + planning foundation; remote execution is work package 3).
+- [ ] 3. Execute source installation through the existing SSH transport.
+- [ ] 4. Add atomic build/install, idempotency, and failure rollback.
+- [ ] 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
+- [ ] 6. Add CentOS-family support.
+- [ ] 7. Plan native Windows service and macOS launchd installers separately.
 
 ## Acceptance Gates
 
-- All three initial Linux images install from a clean state through OpenSSH.
-- The built worker reports the expected version/commit and serves `/healthz`.
-- Re-running the installer succeeds without duplicate services or leaked files.
-- Host-key, checksum, clone, build, and service-start failure tests preserve the
-  previous installation.
-- CI uses approved registry mirrors and cleans every test container/network.
+- [ ] All three initial Linux images install from a clean state through OpenSSH.
+- [ ] The built worker reports the expected version/commit and serves `/healthz`.
+- [ ] Re-running the installer succeeds without duplicate services or leaked files.
+- [ ] Host-key, checksum, clone, build, and service-start failure tests preserve the
+      previous installation.
+- [ ] CI uses approved registry mirrors and cleans every test container/network.
+
+## Verified Test Evidence (work packages 1 and 2)
+
+Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
+
+- Alpine `3.24.1` (mirror `reg.rsxx.ru/library/alpine:3`): distro alpine,
+  pkg apk, init openrc.
+- Ubuntu `24.04` (`VERSION_ID=24.04`): distro ubuntu, pkg apt, init systemd.
+- Arch rolling image `archlinux:latest` (`VERSION_ID=20260809.0.570793`):
+  distro arch, pkg pacman, init systemd.
+- Host-key mismatch, host-key stability, failed-start cleanup, and complete
+  teardown (container, network, fixture image tag, and temp dir gone) tests
+  pass; no test container, network, or image tag is left behind.