worker/docs/source-installation.md

# SSH Source Installation Plan

## Status

In progress. Work package 1 (reusable Docker/OpenSSH harness and distro
fixtures) and the pure detection/planning foundation (work package 2
core) are implemented:

- `internal/installer/harness` builds and runs real OpenSSH containers
  for Alpine, Ubuntu, and Arch, waits for real network readiness, captures
  the server host key into a temp `known_hosts` file, and tears the
  container, network, per-instance fixture image tag, and temp dir down
  reliably. It never mocks SSH and connects with the
  `golang.org/x/crypto/ssh` library and known_hosts verification
  semantics the installer's `deploy` path relies on.
- `internal/sshinstall` resolves a remote host's distro, package manager,
  and init system from `/etc/os-release`, plans the pinned Go 1.26
  toolchain (published SHA-256) for the remote architecture, and produces
  a pure source-install plan. It executes nothing.

The current Go installer can still only upload a binary or deploy an
immutable Docker image over SSH. Remote package/toolchain/source build
execution (work packages 3 and 4) is not implemented yet; the acceptance
test currently stops after detection, clean-state, and planning
assertions. Existing tests are unit tests; the harness tests run against
live OpenSSH containers when explicitly enabled.

## Initial Platform Scope

The first source installer supports Linux and is validated on Alpine, Ubuntu,
and Arch Linux containers. CentOS-family support follows after its package and
service differences are implemented. Windows and macOS remain later platform
work despite the worker being written in Go.

Use `reg.rsxx.ru` image mirrors where available. Tests must not depend on Docker
Hub when a local mirror exists.

## Source Install Flow

The Go CLI connects through `golang.org/x/crypto/ssh` using existing key,
passphrase, password, sudo-password, known-hosts, and pinned-fingerprint support.
It then:

1. detects supported OS, architecture, package manager, init system, and
   privilege path;
2. installs only required packages (`git`, CA certificates, download/archive
   tools); it does not install `build-essential` or a C compiler unless a
   detected dependency requires CGO;
3. downloads the pinned Go 1.26 toolchain for the detected architecture and
   verifies the published SHA-256;
4. clones `https://rocketgit.ru/rsmon/worker.git` or updates an existing clone;
5. checks out branch `main` and records the resolved commit;
6. builds a reproducible worker binary with the repository build flags;
7. atomically installs the binary, validated environment, data directory, and
   service definition;
8. starts the service and verifies process status and `/healthz`.

Repository, branch, Go version, checksum source, build directory, and Go module
proxy may be configurable, but production output records their resolved values.
The default repository is publicly readable and requires no source credential.

## Docker OpenSSH Test Harness

Adapt the real-network pattern from `/data/_swap/sshkeymanager`: start an
OpenSSH container, wait for SSH readiness, connect with the Go installer, and
tear the environment down reliably. Do not mock SSH command execution in the
acceptance test.

Provide images/fixtures for:

- Ubuntu with apt and systemd-compatible service testing where practical;
- Alpine with apk/OpenRC or a clearly separated no-service build/install gate;
- Arch with pacman and its service behavior.

Each clean target begins without Go or the worker source. The test asserts
package installation, verified Go version, clone branch/resolved commit, build,
atomic config permissions, running service where supported, and HTTP liveness.

### Implementation (`internal/installer/harness`)

The harness starts one container per distro fixture
(`testdata/fixtures/{alpine,ubuntu,arch}/Dockerfile`), waits for a real
TCP + SSH handshake, captures the server host key into a temp
`known_hosts` file, and connects with the `golang.org/x/crypto/ssh`
library and known_hosts verification semantics the installer's `deploy`
uses. The harness owns its connection code rather than calling into the
installer package, so the tests stay independent; only the library and
the verification semantics are shared.

Host-key handling is trust-on-first-use (TOFU): the fresh temp
`known_hosts` file accepts the first key the server presents. What the
harness proves is that a known_hosts entry carrying a *different* key is
rejected before any command runs (the `TestHarnessHostKeyMismatch` test),
not that a fingerprint is pinned.

Teardown (`docker rm -f` + `docker network rm` + per-instance
`docker image rm` + temp-dir removal) is idempotent, runs on every
`Start` error path, and is verified by a dedicated test. Each harness
instance builds its own uniquely-tagged fixture image
(`rsmon-worker-test/<fixture>-<suffix>:local`), so removing one never
deletes a shared base image or another instance's image.

Base images, mirror-first:

| Fixture | Default image | Note |
| --- | --- | --- |
| alpine | `reg.rsxx.ru/library/alpine:3` | reg.rsxx.ru mirror exists |
| ubuntu | `ubuntu:24.04` | no mirror yet; override `RSMON_TEST_IMAGE_UBUNTU` |
| arch | `archlinux:latest` | no mirror yet; override `RSMON_TEST_IMAGE_ARCH` |

Any `RSMON_TEST_IMAGE_<NAME>` environment variable overrides the fixture
image, so a mirror or local cache can be used when available.

> **Mutable test images.** The fixture base tags above are deliberately
> mutable (a major-tag mirror ref and Docker Hub rolling tags) so the
> fixtures track current distro releases. Fixture builds are therefore
> not byte-reproducible; the worker's *source-install* production output
> pins immutable artifacts (a Go toolchain SHA-256, a branch's resolved
> commit) and this harness's image override is the escape hatch for
> reproducing a specific distro snapshot.

The fixtures authenticate with the bundled test key
(`testdata/keys/rsmon_test_ed25519`); password auth is disabled and
`PermitRootLogin` is `prohibit-password`. `openrc` is installed in the
Alpine fixture so init detection has a stable marker; Ubuntu and Arch
carry systemd markers.

> **Test-only key.** The bundled keypair is strictly a test fixture: it
> grants root SSH access only to the disposable containers that bake its
> public key. It must never be used for real hosts, added to production
> images, or treated as a credential outside the harness.

### Opt-in integration test controls

Ordinary unit runs never pull or start Docker. The Docker/OpenSSH tests
are gated behind the `RSMON_TEST_DOCKER` environment variable:

- `make test` (default CI unit run) pins `RSMON_TEST_DOCKER=0` and skips
  every harness Docker test, even when the flag is exported in the
  developer's environment.
- `make test-ssh` sets `RSMON_TEST_DOCKER=1` and runs the full fixture
  matrix (`TestHarnessFixtures` for alpine/ubuntu/arch, host-key
  mismatch, stable host key, failed-start cleanup, and teardown
  assertions).
- The harness `Start` itself refuses to run without the opt-in flag.

Run them locally with:

```bash
make test-ssh
```

or, equivalently:

```bash
RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harness
```

## Idempotency And Security

- A second run updates/fetches safely and leaves one active service.
- Wrong host fingerprints fail before remote mutation. The harness's fresh
  known_hosts file is trust-on-first-use; its dedicated mismatch test dials
  against a known_hosts entry carrying a different server key and proves the
  dial fails before any command runs.
- Tokens/passwords come from files or stdin-safe channels and never appear in
  command arguments, logs, source checkout, or shell history.
- Remote temporary files are removed on success and failure.
- Failed builds do not replace a working binary or service definition.
- Package-manager and download failures return bounded actionable errors.
- The installer verifies Go tarball checksum before extraction.

## Implementation Work Packages

- [x] 1. Add reusable Docker/OpenSSH harness and distro fixtures.
- [x] 2. Add pure distro/toolchain/source-install script planning and unit tests
      (detection + planning foundation; remote execution is work package 3).
- [ ] 3. Execute source installation through the existing SSH transport.
- [ ] 4. Add atomic build/install, idempotency, and failure rollback.
- [ ] 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
- [ ] 6. Add CentOS-family support.
- [ ] 7. Plan native Windows service and macOS launchd installers separately.

## Acceptance Gates

- [ ] All three initial Linux images install from a clean state through OpenSSH.
- [ ] The built worker reports the expected version/commit and serves `/healthz`.
- [ ] Re-running the installer succeeds without duplicate services or leaked files.
- [ ] Host-key, checksum, clone, build, and service-start failure tests preserve the
      previous installation.
- [ ] CI uses approved registry mirrors and cleans every test container/network.

## Verified Test Evidence (work packages 1 and 2)

Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):

- Alpine `3.24.1` (mirror `reg.rsxx.ru/library/alpine:3`): distro alpine,
  pkg apk, init openrc.
- Ubuntu `24.04` (`VERSION_ID=24.04`): distro ubuntu, pkg apt, init systemd.
- Arch rolling image `archlinux:latest` (`VERSION_ID=20260809.0.570793`):
  distro arch, pkg pacman, init systemd.
- Host-key mismatch, host-key stability, failed-start cleanup, and complete
  teardown (container, network, fixture image tag, and temp dir gone) tests
  pass; no test container, network, or image tag is left behind.