feat(installer): build worker source over SSH

docs/changelog.md

@@ -2,6 +2,88 @@
 
 ## 2026-08-12
 
+### Source-install hardening review
+
+- Fail-closed remote scripts: checkout, branch resolution, and build steps now
+  run under `set -eu` (and package/record steps chain with `&&`), so a failed
+  checkout or fetch can never be masked by a stale `rev-parse` or subsequent
+  command. The checkout step additionally refuses (`git diff --quiet` /
+  `--cached --quiet`) before the destructive `checkout -B`, because `-B`
+  silently discards local changes and would otherwise never fail on a dirty
+  tree. A dirty-tree checkout failure surfaces as a `check out branch` error
+  before the build runs; the new `TestSourceInstallSSHCheckoutFailureNotMasked`
+  unit test and `TestSourceInstallDirtyCheckoutPreservesStaging` Docker test
+  prove the previous staging binary and commit record are preserved
+  byte-for-byte.
+- Atomic toolchain replacement: the Go toolchain is downloaded, SHA-256
+  verified, extracted into a same-filesystem staging dir, verified to report
+  the target version, and only then swapped into `ToolchainDir` with the prior
+  toolchain moved to a sibling `.go-backup` that is restored on swap failure.
+  A failed download/verify/extract/swap never destroys the prior Go.
+- Record-after-build pairing: `rsmon-worker.commit` is written only after a
+  successful build, so the record and the staged binary always correspond to
+  the same commit. The build verifies `<stage>.new --version` before an atomic
+  `mv -f` over the previous staging binary; `GOMODCACHE` is now set alongside
+  `GOCACHE` inside the build dir so reruns reuse both caches.
+- Origin verification: an existing checkout's `remote.origin.url` must exactly
+  match the configured repository before anything is fetched or built.
+- Repository hardening: only `https://` clone URLs without userinfo are
+  accepted (`ValidateRepoURL`, enforced before dialing and again when
+  planning).
+- Explicit charset validation for Go version and architecture overrides
+  (`sshinstall.ValidGoVersion` / `ValidGoArch`) before any remote mutation.
+- Bounded remote execution: each remote command is capped by
+  `--session-timeout` (default 30m) and captured stdout is size-bounded
+  alongside the existing stderr bound; deploy's streaming `runRemote` keeps its
+  historical no-timeout behavior.
+- The source installer now defaults to the remote's default branch (the public
+  repo publishes `master`) instead of the plan's stale `main` default, while
+  `--branch` still pins an explicit branch that must exist remotely. The
+  README quickstart no longer shows the incorrect `--branch main`.
+- CLI secret flags keep their compatibility, but docs now explicitly state that
+  file options (`-password-file`, etc.) keep secrets out of argv and shell
+  history while direct flags expose them through the process list.
+- The harness accepts `RSMON_TEST_DOCKER_DNS` (comma-separated) to pin
+  `docker run --dns` for fixture containers, so internet-facing installs are
+  not at the mercy of a flaky local resolver.
+
+### Remote source-install execution (work package 3)
+
+- Added `installer.SourceInstall` (`internal/installer/sourceinstall.go`):
+  executes the source-install flow through the existing SSH transport,
+  reusing the `deploy` command's `SSHOptions` (keys, passphrases,
+  passwords, sudo passwords, known-hosts, pinned fingerprints) and its
+  privilege path. Extracted the shared `SSHOptions` struct and a
+  `sudoWrap` helper so deploy and source install cannot diverge.
+- Steps implemented: minimal package-prerequisite install per distro
+  (`apk`/`apt`/`pacman`/`dnf`, never a compiler), SHA-256-verified Go 1.26
+  toolchain download/extraction with an idempotent version-skip and temp-dir
+  cleanup, clone-or-update of the public repository (with a bounded 3-attempt
+  retry for transient DNS/TLS/proxy failures), resolution of the remote
+  default branch (a pinned branch must exist remotely), a resolved branch and
+  commit record at `<BuildDir>/rsmon-worker.commit`, and a staging build
+  (`CGO_ENABLED=0`, `-trimpath`, repository `-ldflags`) verified via
+  `--version`. The running service, config, and data directory are untouched
+  (work package 4 boundary).
+- Security: every interpolated remote value is single-quoted; branch and
+  commit values are strictly validated; no worker token or control-plane
+  credential is sent; sudo passwords travel only over session stdin; remote
+  errors are bounded (stderr truncated in `runRemoteOutput`).
+- Added unit tests for the remote scripts, option validation, branch/commit
+  parsing, sudo wrapping, the secrets-absent contract, and an in-process
+  real-SSH orchestration flow (with missing-pinned-branch, build-failure, and
+  detection-failure paths).
+- Added `TestSourceInstallFixtures` to the Docker/OpenSSH harness: each of
+  Alpine, Ubuntu, and Arch installs from a clean state through the real
+  harness transport (prerequisite install, verified Go 1.26, clone, resolved
+  commit, staging build), then a rerun proves idempotency (same branch,
+  toolchain reuse, no temp leaks). All three resolved the public repo's
+  `master` at `4651deb2...` in the recorded run. `make test-ssh` timeout
+  raised to 60m.
+- Documented the branch-resolution reality: the public repository currently
+  publishes `master`, and the installer records whatever the remote default
+  branch resolves to.
+
 ### Source-install harness and planning foundations (work packages 1-2)
 
 - Added `internal/installer/harness`: a reusable Docker/OpenSSH test harness

docs/implementation-roadmap.md

@@ -15,8 +15,8 @@ Worker repository:
   metadata;
 - add a package/install smoke test for Docker and systemd artifacts;
 - add the Docker/OpenSSH source-install matrix for Alpine, Ubuntu, and Arch from
-  [source-installation.md](source-installation.md), using Go 1.26 and branch
-  `main`;
+  [source-installation.md](source-installation.md), using Go 1.26 and the
+  remote default branch;
 - document immutable SHA and release tags as production defaults.
 
 Source-install foundations landed:
@@ -26,9 +26,16 @@ Source-install foundations landed:
       reliable teardown, gated behind `RSMON_TEST_DOCKER` (`make test-ssh`);
 - [x] pure distro/toolchain/source-install planning (`internal/sshinstall`):
       os-release detection, package-manager/init resolution, pinned Go 1.26
-      toolchain with published SHA-256, and a plan the executor will run;
-- [ ] remote package install, Go download, clone, and build execution through
-      the SSH transport (source-install work package 3);
+      toolchain with published SHA-256, and a plan the executor runs;
+- [x] remote package install, Go download, clone, and build execution through
+      the SSH transport (`installer.SourceInstall`): prerequisite install,
+      SHA-256-verified Go toolchain, clone/update of the public repo, resolved
+      branch/commit record, and a staging build. Running service/config is not
+      touched (source-install work package 3);
+- [ ] atomic service activation, rollback, and failure-preservation tests over
+      SSH (source-install work package 4);
+- [ ] run the full source-install E2E matrix in CI (source-install work
+      package 5).
 
 Gate: a push publishes `sha-<12>` and `latest` manifests for both platforms,
 and a container remains healthy when the control plane is unavailable.

docs/install.md

@@ -5,8 +5,12 @@ Docker image) into a running, enabled systemd service on a Linux host. It is
 the supported way to deploy the worker: it writes the configuration, the
 systemd unit, and the data directory, then starts the service.
 
-> A Go SSH source installer is planned in
-> [source-installation.md](source-installation.md). Today
+> A Go SSH source installer (`rsmon-worker source-install`) builds the worker
+> from source on a remote host over SSH - prerequisites, verified Go toolchain,
+> clone/update of the public repository (resolved to the remote default branch
+> unless pinned), resolved branch/commit record, and a staging build - without
+> yet installing a service. See
+> [source-installation.md](source-installation.md). Today the local
 > `install` copies the binary you invoke it from (or pulls the `--image`
 > digest), so build first with `make build` and run the resulting
 > `./bin/rsmon-worker`.

docs/source-installation.md

@@ -3,8 +3,9 @@
 ## Status
 
 In progress. Work package 1 (reusable Docker/OpenSSH harness and distro
-fixtures) and the pure detection/planning foundation (work package 2
-core) are implemented:
+fixtures), the pure detection/planning foundation (work package 2 core),
+and work package 3 (remote execution through the existing SSH transport)
+are implemented:
 
 - `internal/installer/harness` builds and runs real OpenSSH containers
   for Alpine, Ubuntu, and Arch, waits for real network readiness, captures
@@ -17,13 +18,23 @@ core) are implemented:
   and init system from `/etc/os-release`, plans the pinned Go 1.26
   toolchain (published SHA-256) for the remote architecture, and produces
   a pure source-install plan. It executes nothing.
+- `installer.SourceInstall` (work package 3) executes the plan through
+  the same SSH transport, authentication, and host-key verification the
+  `deploy` command uses. It installs the minimal package prerequisites,
+  downloads and SHA-256-verifies the pinned Go toolchain before
+  extraction, clones/updates the public repository, checks out the
+  resolved branch, records the resolved branch and commit, and builds the
+  worker to a staging path. It deliberately does not install or replace
+  the running service, configuration, or data directory: atomic
+  activation and rollback are the next work package.
 
 The current Go installer can still only upload a binary or deploy an
-immutable Docker image over SSH. Remote package/toolchain/source build
-execution (work packages 3 and 4) is not implemented yet; the acceptance
-test currently stops after detection, clean-state, and planning
-assertions. Existing tests are unit tests; the harness tests run against
-live OpenSSH containers when explicitly enabled.
+immutable Docker image over SSH. Source installs now build remotely to a
+staging path, but service activation over SSH (work package 4) is not
+implemented yet; the acceptance test stops after install-to-staging and
+idempotent-rerun assertions. Existing tests are unit tests plus the
+harness tests that run against live OpenSSH containers when explicitly
+enabled.
 
 ## Initial Platform Scope
 
@@ -49,7 +60,8 @@ It then:
 3. downloads the pinned Go 1.26 toolchain for the detected architecture and
    verifies the published SHA-256;
 4. clones `https://rocketgit.ru/rsmon/worker.git` or updates an existing clone;
-5. checks out branch `main` and records the resolved commit;
+5. checks out the resolved branch (the pinned branch when one is configured,
+   otherwise the remote's default branch) and records the resolved commit;
 6. builds a reproducible worker binary with the repository build flags;
 7. atomically installs the binary, validated environment, data directory, and
    service definition;
@@ -59,6 +71,100 @@ Repository, branch, Go version, checksum source, build directory, and Go module
 proxy may be configurable, but production output records their resolved values.
 The default repository is publicly readable and requires no source credential.
 
+## Work Package 3: Remote Execution To A Staging Path
+
+Work package 3 is `installer.SourceInstall` in
+`internal/installer/sourceinstall.go`. It reuses the `deploy` command's
+`SSHOptions` (authentication, sudo password, known-hosts and fingerprint
+verification) and runs every remote step with the same privilege path
+(root, passwordless sudo, or `sudo -S -p ''` with the password delivered
+only over stdin). Steps 1-6 of the flow above are implemented; step 7
+(atomic install) is deliberately the next work package.
+
+Per step:
+
+- **Prerequisite install.** `packageScript` renders the distro's
+  idempotent command (`apk add --no-cache`, `apt-get update` +
+  `apt-get install -y --no-install-recommends`, `pacman -Sy --noconfirm
+  --needed`, `dnf install -y`) for the minimal plan packages (`git`,
+  `ca-certificates`, `curl`, `tar`, `gzip`). No compiler is ever planned
+  or installed.
+- **Toolchain.** `toolchainScript` downloads the pinned Go tarball into a
+  `mktemp` temp dir, verifies it with `sha256sum -c -` *before*
+  extraction, extracts into a staging dir on the same filesystem as
+  `ToolchainDir` (which must end in `/go`, default `/usr/local/go`),
+  verifies the staged toolchain reports the target version, and only then
+  swaps it into place. The prior toolchain is moved to a sibling
+  `.go-backup` and is restored if the swap fails, so a failed
+  download/verify/extract/swap always leaves the prior Go untouched. A
+  present toolchain that already reports the target version is reused, so
+  reruns do not re-download. Temp, staging, and backup directories are
+  removed on success and failure.
+- **Clone/update.** `cloneUpdateScript` clones the repository when
+  `BuildDir` has no `.git` and otherwise fetches with `--prune`, so a
+  rerun updates in place. An existing checkout's `remote.origin.url` must
+  exactly match the configured repository before anything is fetched or
+  built, so the installer can never fetch or build an unconfigured
+  repository. The clone/fetch retries up to three times (2s apart)
+  because real repositories can be transiently unreachable (DNS, TLS, or
+  proxy hiccups); three bounded attempts keep a momentary outage from
+  failing a full source install.
+- **Resolved branch.** The installer resolves the remote default branch
+  via `git remote set-head origin --auto` +
+  `git symbolic-ref --short refs/remotes/origin/HEAD`. When no branch is
+  pinned it builds the remote default (the public repo currently
+  publishes `master`); a pinned branch must exist remotely or the install
+  fails before the build. The resolved branch and the `git rev-parse
+  HEAD` commit (validated as 40 lowercase hex) are returned by
+  `SourceInstall`.
+- **Staging build.** `buildScript` builds with `CGO_ENABLED=0`,
+  `-trimpath`, the repository's own `-ldflags` shape (version `dev`,
+  resolved commit short form, UTC build date), and both `GOCACHE` and
+  `GOMODCACHE` inside the build dir (so reruns reuse them), plus an
+  optional `GOPROXY`. The binary is built to a sibling `<stage>.new`,
+  verified with `<stage>.new --version`, and only then atomically swapped
+  over `<BuildDir>/rsmon-worker` (or `StageBinary`), so a failed build
+  never replaces the previous staging binary. It is not written to
+  `/usr/local/bin`.
+- **Commit record.** The `rsmon-worker.commit` record (a
+  `branch=...` / `commit=...` format in the build dir) is written only
+  *after* a successful build, so the record and the staged binary always
+  correspond to the same commit.
+
+Security properties of work package 3:
+
+- Every interpolated value (repository, branch, build dir, URLs, SHA-256,
+  package names, paths) is single-quoted; repository, branch, commit, Go
+  version, and Go architecture values are additionally validated with
+  strict patterns. No worker token or control-plane credential is ever
+  sent: the install stages a binary and touches no service configuration.
+- The repository must be an `https://` URL without userinfo, so source
+  credentials cannot reach the remote clone command or the clone's
+  config.
+- Sudo passwords are delivered over the session's stdin only, never in a
+  command string (the same `sudoWrap` path the `deploy` command uses).
+  Direct `--password`/`--sudo-password`/`--key-passphrase` flags remain
+  available but expose the value through the process list and shell
+  history; the CLI docs strongly prefer the `-file` variants. The source
+  installer sends no worker token at all.
+- Every step script fails closed: `set -eu` (or an explicit retry that
+  exits non-zero) is used, so a failed checkout or fetch can never be
+  masked by a stale subsequent command. The checkout step refuses before
+  the destructive `checkout -B` when the tracked working tree is dirty
+  (`git diff --quiet` / `--cached --quiet`), because `checkout -B` would
+  silently discard local changes; a dirty-tree rerun fails at checkout and
+  leaves the previous staging binary and commit record untouched.
+- Remote errors are bounded: each step returns a step-labelled error,
+  stderr and captured stdout are size-bounded in `runRemoteOutput`, and
+  each remote command is capped by `--session-timeout` (default 30m).
+- Toolchain temp, staging, and backup directories are removed on success
+  and failure, and the acceptance test asserts no `/tmp/rsmon-toolchain-*`
+  or `/usr/local/.go-staging-*`/`.go-backup` leaks after the rerun.
+- Failed builds and failed checkouts leave the previous staging binary
+  untouched (the binary is only overwritten by an atomically-swapped
+  successful build, and a checkout failure aborts before the build);
+  there is no running service or configuration to preserve yet.
+
 ## Docker OpenSSH Test Harness
 
 Adapt the real-network pattern from `/data/_swap/sshkeymanager`: start an
@@ -93,6 +199,11 @@ harness proves is that a known_hosts entry carrying a *different* key is
 rejected before any command runs (the `TestHarnessHostKeyMismatch` test),
 not that a fingerprint is pinned.
 
+The fixtures install over the public internet, so environments with flaky
+local resolvers can pin a reliable upstream via the comma-separated
+`RSMON_TEST_DOCKER_DNS` variable (applied as `docker run --dns ...`); it
+is empty by default, keeping Docker's embedded DNS.
+
 Teardown (`docker rm -f` + `docker network rm` + per-instance
 `docker image rm` + temp-dir removal) is idempotent, runs on every
 `Start` error path, and is verified by a dedicated test. Each harness
@@ -158,15 +269,26 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
 
 ## Idempotency And Security
 
-- A second run updates/fetches safely and leaves one active service.
+- A second run updates/fetches safely: the toolchain is reused when the version
+  matches, the clone's origin is verified against the configured repository and
+  then fetched in place, the resolved branch/commit record is rewritten after
+  the new build succeeds, and the staging build atomically swaps over the
+  previous staging binary. The "one active service" guarantee is a work package
+  4 property (activation); work package 3 leaves no running service to
+  duplicate and no leaked toolchain temp files.
 - Wrong host fingerprints fail before remote mutation. The harness's fresh
   known_hosts file is trust-on-first-use; its dedicated mismatch test dials
   against a known_hosts entry carrying a different server key and proves the
   dial fails before any command runs.
-- Tokens/passwords come from files or stdin-safe channels and never appear in
-  command arguments, logs, source checkout, or shell history.
+- Tokens/passwords should come from files: `--key-passphrase-file`,
+  `--password-file`, and `--sudo-password-file` keep secrets out of argv and
+  shell history, while the equivalent direct flags expose them through the
+  process list. The source install itself sends no worker token or
+  control-plane credential at all, and sudo passwords travel only over the
+  session's stdin.
 - Remote temporary files are removed on success and failure.
-- Failed builds do not replace a working binary or service definition.
+- Failed builds, failed checkouts, and failed toolchain swaps do not replace a
+  working binary or service definition.
 - Package-manager and download failures return bounded actionable errors.
 - The installer verifies Go tarball checksum before extraction.
 
@@ -175,7 +297,9 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
 - [x] 1. Add reusable Docker/OpenSSH harness and distro fixtures.
 - [x] 2. Add pure distro/toolchain/source-install script planning and unit tests
       (detection + planning foundation; remote execution is work package 3).
-- [ ] 3. Execute source installation through the existing SSH transport.
+- [x] 3. Execute source installation through the existing SSH transport
+      (prerequisites, verified Go toolchain, clone/update, resolved branch and
+      commit record, and a build to a staging path; no service activation).
 - [ ] 4. Add atomic build/install, idempotency, and failure rollback.
 - [ ] 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
 - [ ] 6. Add CentOS-family support.
@@ -184,13 +308,16 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
 ## Acceptance Gates
 
 - [ ] All three initial Linux images install from a clean state through OpenSSH.
+      Work package 3 covers the install-to-staging half (package install,
+      verified Go 1.26, clone/update, resolved commit, staging build); the
+      service-start half is the work package 4 gate.
 - [ ] The built worker reports the expected version/commit and serves `/healthz`.
 - [ ] Re-running the installer succeeds without duplicate services or leaked files.
 - [ ] Host-key, checksum, clone, build, and service-start failure tests preserve the
       previous installation.
 - [ ] CI uses approved registry mirrors and cleans every test container/network.
 
-## Verified Test Evidence (work packages 1 and 2)
+## Verified Test Evidence (work packages 1-3)
 
 Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
 
@@ -202,3 +329,19 @@ Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
 - Host-key mismatch, host-key stability, failed-start cleanup, and complete
   teardown (container, network, fixture image tag, and temp dir gone) tests
   pass; no test container, network, or image tag is left behind.
+- `TestSourceInstallFixtures` runs the full work-package-3 flow on each
+  fixture from a clean state over the harness's real OpenSSH transport:
+  prerequisite install, verified Go 1.26 toolchain download/extraction, clone
+  of the public repository, resolution of its default branch, resolved-commit
+  record file, and a staging build that reports the resolved commit via
+  `--version`. The public repo's `HEAD` was `master` at `4651deb2...` during
+  the run, and all three fixtures resolved to that branch and commit (the
+  installer records whatever the remote publishes). A rerun succeeds, keeps
+  the same branch, reuses the toolchain, and leaves no
+  `/tmp/rsmon-toolchain-*` temp dirs. The running service and its
+  configuration are not touched (work package 4 boundary).
+- The unit suite covers the per-step remote scripts, option validation,
+  branch/commit parsing, sudo wrapping (password never in the command), the
+  secrets-absent contract, and an in-process real-SSH orchestration flow with
+  failure paths for missing pinned branches, build failures, and detection
+  failures.