feat(installer): build worker source over SSH

docs/source-installation.md

@@ -3,8 +3,9 @@
 ## Status
 
 In progress. Work package 1 (reusable Docker/OpenSSH harness and distro
-fixtures) and the pure detection/planning foundation (work package 2
-core) are implemented:
+fixtures), the pure detection/planning foundation (work package 2 core),
+and work package 3 (remote execution through the existing SSH transport)
+are implemented:
 
 - `internal/installer/harness` builds and runs real OpenSSH containers
   for Alpine, Ubuntu, and Arch, waits for real network readiness, captures
@@ -17,13 +18,23 @@ core) are implemented:
   and init system from `/etc/os-release`, plans the pinned Go 1.26
   toolchain (published SHA-256) for the remote architecture, and produces
   a pure source-install plan. It executes nothing.
+- `installer.SourceInstall` (work package 3) executes the plan through
+  the same SSH transport, authentication, and host-key verification the
+  `deploy` command uses. It installs the minimal package prerequisites,
+  downloads and SHA-256-verifies the pinned Go toolchain before
+  extraction, clones/updates the public repository, checks out the
+  resolved branch, records the resolved branch and commit, and builds the
+  worker to a staging path. It deliberately does not install or replace
+  the running service, configuration, or data directory: atomic
+  activation and rollback are the next work package.
 
 The current Go installer can still only upload a binary or deploy an
-immutable Docker image over SSH. Remote package/toolchain/source build
-execution (work packages 3 and 4) is not implemented yet; the acceptance
-test currently stops after detection, clean-state, and planning
-assertions. Existing tests are unit tests; the harness tests run against
-live OpenSSH containers when explicitly enabled.
+immutable Docker image over SSH. Source installs now build remotely to a
+staging path, but service activation over SSH (work package 4) is not
+implemented yet; the acceptance test stops after install-to-staging and
+idempotent-rerun assertions. Existing tests are unit tests plus the
+harness tests that run against live OpenSSH containers when explicitly
+enabled.
 
 ## Initial Platform Scope
 
@@ -49,7 +60,8 @@ It then:
 3. downloads the pinned Go 1.26 toolchain for the detected architecture and
    verifies the published SHA-256;
 4. clones `https://rocketgit.ru/rsmon/worker.git` or updates an existing clone;
-5. checks out branch `main` and records the resolved commit;
+5. checks out the resolved branch (the pinned branch when one is configured,
+   otherwise the remote's default branch) and records the resolved commit;
 6. builds a reproducible worker binary with the repository build flags;
 7. atomically installs the binary, validated environment, data directory, and
    service definition;
@@ -59,6 +71,100 @@ Repository, branch, Go version, checksum source, build directory, and Go module
 proxy may be configurable, but production output records their resolved values.
 The default repository is publicly readable and requires no source credential.
 
+## Work Package 3: Remote Execution To A Staging Path
+
+Work package 3 is `installer.SourceInstall` in
+`internal/installer/sourceinstall.go`. It reuses the `deploy` command's
+`SSHOptions` (authentication, sudo password, known-hosts and fingerprint
+verification) and runs every remote step with the same privilege path
+(root, passwordless sudo, or `sudo -S -p ''` with the password delivered
+only over stdin). Steps 1-6 of the flow above are implemented; step 7
+(atomic install) is deliberately the next work package.
+
+Per step:
+
+- **Prerequisite install.** `packageScript` renders the distro's
+  idempotent command (`apk add --no-cache`, `apt-get update` +
+  `apt-get install -y --no-install-recommends`, `pacman -Sy --noconfirm
+  --needed`, `dnf install -y`) for the minimal plan packages (`git`,
+  `ca-certificates`, `curl`, `tar`, `gzip`). No compiler is ever planned
+  or installed.
+- **Toolchain.** `toolchainScript` downloads the pinned Go tarball into a
+  `mktemp` temp dir, verifies it with `sha256sum -c -` *before*
+  extraction, extracts into a staging dir on the same filesystem as
+  `ToolchainDir` (which must end in `/go`, default `/usr/local/go`),
+  verifies the staged toolchain reports the target version, and only then
+  swaps it into place. The prior toolchain is moved to a sibling
+  `.go-backup` and is restored if the swap fails, so a failed
+  download/verify/extract/swap always leaves the prior Go untouched. A
+  present toolchain that already reports the target version is reused, so
+  reruns do not re-download. Temp, staging, and backup directories are
+  removed on success and failure.
+- **Clone/update.** `cloneUpdateScript` clones the repository when
+  `BuildDir` has no `.git` and otherwise fetches with `--prune`, so a
+  rerun updates in place. An existing checkout's `remote.origin.url` must
+  exactly match the configured repository before anything is fetched or
+  built, so the installer can never fetch or build an unconfigured
+  repository. The clone/fetch retries up to three times (2s apart)
+  because real repositories can be transiently unreachable (DNS, TLS, or
+  proxy hiccups); three bounded attempts keep a momentary outage from
+  failing a full source install.
+- **Resolved branch.** The installer resolves the remote default branch
+  via `git remote set-head origin --auto` +
+  `git symbolic-ref --short refs/remotes/origin/HEAD`. When no branch is
+  pinned it builds the remote default (the public repo currently
+  publishes `master`); a pinned branch must exist remotely or the install
+  fails before the build. The resolved branch and the `git rev-parse
+  HEAD` commit (validated as 40 lowercase hex) are returned by
+  `SourceInstall`.
+- **Staging build.** `buildScript` builds with `CGO_ENABLED=0`,
+  `-trimpath`, the repository's own `-ldflags` shape (version `dev`,
+  resolved commit short form, UTC build date), and both `GOCACHE` and
+  `GOMODCACHE` inside the build dir (so reruns reuse them), plus an
+  optional `GOPROXY`. The binary is built to a sibling `<stage>.new`,
+  verified with `<stage>.new --version`, and only then atomically swapped
+  over `<BuildDir>/rsmon-worker` (or `StageBinary`), so a failed build
+  never replaces the previous staging binary. It is not written to
+  `/usr/local/bin`.
+- **Commit record.** The `rsmon-worker.commit` record (a
+  `branch=...` / `commit=...` format in the build dir) is written only
+  *after* a successful build, so the record and the staged binary always
+  correspond to the same commit.
+
+Security properties of work package 3:
+
+- Every interpolated value (repository, branch, build dir, URLs, SHA-256,
+  package names, paths) is single-quoted; repository, branch, commit, Go
+  version, and Go architecture values are additionally validated with
+  strict patterns. No worker token or control-plane credential is ever
+  sent: the install stages a binary and touches no service configuration.
+- The repository must be an `https://` URL without userinfo, so source
+  credentials cannot reach the remote clone command or the clone's
+  config.
+- Sudo passwords are delivered over the session's stdin only, never in a
+  command string (the same `sudoWrap` path the `deploy` command uses).
+  Direct `--password`/`--sudo-password`/`--key-passphrase` flags remain
+  available but expose the value through the process list and shell
+  history; the CLI docs strongly prefer the `-file` variants. The source
+  installer sends no worker token at all.
+- Every step script fails closed: `set -eu` (or an explicit retry that
+  exits non-zero) is used, so a failed checkout or fetch can never be
+  masked by a stale subsequent command. The checkout step refuses before
+  the destructive `checkout -B` when the tracked working tree is dirty
+  (`git diff --quiet` / `--cached --quiet`), because `checkout -B` would
+  silently discard local changes; a dirty-tree rerun fails at checkout and
+  leaves the previous staging binary and commit record untouched.
+- Remote errors are bounded: each step returns a step-labelled error,
+  stderr and captured stdout are size-bounded in `runRemoteOutput`, and
+  each remote command is capped by `--session-timeout` (default 30m).
+- Toolchain temp, staging, and backup directories are removed on success
+  and failure, and the acceptance test asserts no `/tmp/rsmon-toolchain-*`
+  or `/usr/local/.go-staging-*`/`.go-backup` leaks after the rerun.
+- Failed builds and failed checkouts leave the previous staging binary
+  untouched (the binary is only overwritten by an atomically-swapped
+  successful build, and a checkout failure aborts before the build);
+  there is no running service or configuration to preserve yet.
+
 ## Docker OpenSSH Test Harness
 
 Adapt the real-network pattern from `/data/_swap/sshkeymanager`: start an
@@ -93,6 +199,11 @@ harness proves is that a known_hosts entry carrying a *different* key is
 rejected before any command runs (the `TestHarnessHostKeyMismatch` test),
 not that a fingerprint is pinned.
 
+The fixtures install over the public internet, so environments with flaky
+local resolvers can pin a reliable upstream via the comma-separated
+`RSMON_TEST_DOCKER_DNS` variable (applied as `docker run --dns ...`); it
+is empty by default, keeping Docker's embedded DNS.
+
 Teardown (`docker rm -f` + `docker network rm` + per-instance
 `docker image rm` + temp-dir removal) is idempotent, runs on every
 `Start` error path, and is verified by a dedicated test. Each harness
@@ -158,15 +269,26 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
 
 ## Idempotency And Security
 
-- A second run updates/fetches safely and leaves one active service.
+- A second run updates/fetches safely: the toolchain is reused when the version
+  matches, the clone's origin is verified against the configured repository and
+  then fetched in place, the resolved branch/commit record is rewritten after
+  the new build succeeds, and the staging build atomically swaps over the
+  previous staging binary. The "one active service" guarantee is a work package
+  4 property (activation); work package 3 leaves no running service to
+  duplicate and no leaked toolchain temp files.
 - Wrong host fingerprints fail before remote mutation. The harness's fresh
   known_hosts file is trust-on-first-use; its dedicated mismatch test dials
   against a known_hosts entry carrying a different server key and proves the
   dial fails before any command runs.
-- Tokens/passwords come from files or stdin-safe channels and never appear in
-  command arguments, logs, source checkout, or shell history.
+- Tokens/passwords should come from files: `--key-passphrase-file`,
+  `--password-file`, and `--sudo-password-file` keep secrets out of argv and
+  shell history, while the equivalent direct flags expose them through the
+  process list. The source install itself sends no worker token or
+  control-plane credential at all, and sudo passwords travel only over the
+  session's stdin.
 - Remote temporary files are removed on success and failure.
-- Failed builds do not replace a working binary or service definition.
+- Failed builds, failed checkouts, and failed toolchain swaps do not replace a
+  working binary or service definition.
 - Package-manager and download failures return bounded actionable errors.
 - The installer verifies Go tarball checksum before extraction.
 
@@ -175,7 +297,9 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
 - [x] 1. Add reusable Docker/OpenSSH harness and distro fixtures.
 - [x] 2. Add pure distro/toolchain/source-install script planning and unit tests
       (detection + planning foundation; remote execution is work package 3).
-- [ ] 3. Execute source installation through the existing SSH transport.
+- [x] 3. Execute source installation through the existing SSH transport
+      (prerequisites, verified Go toolchain, clone/update, resolved branch and
+      commit record, and a build to a staging path; no service activation).
 - [ ] 4. Add atomic build/install, idempotency, and failure rollback.
 - [ ] 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
 - [ ] 6. Add CentOS-family support.
@@ -184,13 +308,16 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
 ## Acceptance Gates
 
 - [ ] All three initial Linux images install from a clean state through OpenSSH.
+      Work package 3 covers the install-to-staging half (package install,
+      verified Go 1.26, clone/update, resolved commit, staging build); the
+      service-start half is the work package 4 gate.
 - [ ] The built worker reports the expected version/commit and serves `/healthz`.
 - [ ] Re-running the installer succeeds without duplicate services or leaked files.
 - [ ] Host-key, checksum, clone, build, and service-start failure tests preserve the
       previous installation.
 - [ ] CI uses approved registry mirrors and cleans every test container/network.
 
-## Verified Test Evidence (work packages 1 and 2)
+## Verified Test Evidence (work packages 1-3)
 
 Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
 
@@ -202,3 +329,19 @@ Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
 - Host-key mismatch, host-key stability, failed-start cleanup, and complete
   teardown (container, network, fixture image tag, and temp dir gone) tests
   pass; no test container, network, or image tag is left behind.
+- `TestSourceInstallFixtures` runs the full work-package-3 flow on each
+  fixture from a clean state over the harness's real OpenSSH transport:
+  prerequisite install, verified Go 1.26 toolchain download/extraction, clone
+  of the public repository, resolution of its default branch, resolved-commit
+  record file, and a staging build that reports the resolved commit via
+  `--version`. The public repo's `HEAD` was `master` at `4651deb2...` during
+  the run, and all three fixtures resolved to that branch and commit (the
+  installer records whatever the remote publishes). A rerun succeeds, keeps
+  the same branch, reuses the toolchain, and leaves no
+  `/tmp/rsmon-toolchain-*` temp dirs. The running service and its
+  configuration are not touched (work package 4 boundary).
+- The unit suite covers the per-step remote scripts, option validation,
+  branch/commit parsing, sudo wrapping (password never in the command), the
+  secrets-absent contract, and an in-process real-SSH orchestration flow with
+  failure paths for missing pinned branches, build failures, and detection
+  failures.