feat(installer): activate source builds atomically

docs/source-installation.md

@@ -2,10 +2,7 @@
 
 ## Status
 
-In progress. Work package 1 (reusable Docker/OpenSSH harness and distro
-fixtures), the pure detection/planning foundation (work package 2 core),
-and work package 3 (remote execution through the existing SSH transport)
-are implemented:
+In progress. Work packages 1-4 are implemented:
 
 - `internal/installer/harness` builds and runs real OpenSSH containers
   for Alpine, Ubuntu, and Arch, waits for real network readiness, captures
@@ -24,17 +21,21 @@ are implemented:
   downloads and SHA-256-verifies the pinned Go toolchain before
   extraction, clones/updates the public repository, checks out the
   resolved branch, records the resolved branch and commit, and builds the
-  worker to a staging path. It deliberately does not install or replace
-  the running service, configuration, or data directory: atomic
-  activation and rollback are the next work package.
+  worker to a staging path.
+- Work package 4 (`installer.SourceInstall` with `Activation` enabled)
+  atomically installs the staged binary, validated environment, data
+  directory, and the detected init's service definition; starts/restarts
+  the worker; verifies the process and `/healthz`; and rolls back to the
+  prior working install on any activation/start/health failure. Reruns
+  are idempotent with exactly one running worker and no leaked temp
+  files. The CLI runs the full flow by default and accepts
+  `--no-activate` (staging only) and `--no-start`.
 
-The current Go installer can still only upload a binary or deploy an
-immutable Docker image over SSH. Source installs now build remotely to a
-staging path, but service activation over SSH (work package 4) is not
-implemented yet; the acceptance test stops after install-to-staging and
-idempotent-rerun assertions. Existing tests are unit tests plus the
-harness tests that run against live OpenSSH containers when explicitly
-enabled.
+The current Go installer can also upload a binary or deploy an immutable
+Docker image over SSH (`deploy`). Source installs build remotely, then
+activate the staged build atomically with rollback. Existing tests are
+unit tests plus the harness tests that run against live OpenSSH
+containers when explicitly enabled.
 
 ## Initial Platform Scope
 
@@ -66,7 +67,6 @@ It then:
 7. atomically installs the binary, validated environment, data directory, and
    service definition;
 8. starts the service and verifies process status and `/healthz`.
-
 Repository, branch, Go version, checksum source, build directory, and Go module
 proxy may be configurable, but production output records their resolved values.
 The default repository is publicly readable and requires no source credential.
@@ -79,7 +79,7 @@ Work package 3 is `installer.SourceInstall` in
 verification) and runs every remote step with the same privilege path
 (root, passwordless sudo, or `sudo -S -p ''` with the password delivered
 only over stdin). Steps 1-6 of the flow above are implemented; step 7
-(atomic install) is deliberately the next work package.
+(atomic install) and step 8 (start + verify) are work package 4.
 
 Per step:
 
@@ -162,8 +162,89 @@ Security properties of work package 3:
   or `/usr/local/.go-staging-*`/`.go-backup` leaks after the rerun.
 - Failed builds and failed checkouts leave the previous staging binary
   untouched (the binary is only overwritten by an atomically-swapped
-  successful build, and a checkout failure aborts before the build);
-  there is no running service or configuration to preserve yet.
+  successful build, and a checkout failure aborts before the build).
+
+## Work Package 4: Atomic Activation And Rollback
+
+Work package 4 is the second half of `installer.SourceInstall`, driven by
+`Activation` (`internal/installer/sourceactivate.go`). It runs over the
+same SSH executor, privilege path, and bounded-error machinery as the
+staging steps and installs the staged build atomically:
+
+- **Layout.** The classic installer's on-disk layout is reused
+  (`resolvePaths`): binary `/usr/local/bin/rsmon-worker[-name]`, config
+  `/etc/rsmon-worker[-name]/worker.env` (mode 0600), data
+  `/var/lib/rsmon-worker[-name]`, and a service definition for the
+  detected init (systemd unit `/etc/systemd/system/rsmon-worker.service`
+  mode 0644, or an OpenRC script `/etc/init.d/rsmon-worker` mode 0755).
+  A named instance (`--name`) mirrors the classic multi-instance
+  convention with `-<name>` suffixes and its own unit and port.
+- **Environment.** The env file is rendered by the classic installer's
+  `resolveInstallEnv` + `renderEnvFile` (so `PUBLIC_URL` canonicalization,
+  basic-auth XOR, required `RSMON_URL`/`RSMON_TOKEN`, and systemd-safe
+  value validation are identical to `install`). The env file is read
+  *exactly once* and validated from the in-memory bytes
+  (`parseEnvironmentContent`), and the rendered env is cached during
+  option normalization and reused at activation time, so a hostile local
+  writer cannot swap the file between the preflight and the remote
+  install (env-file TOCTOU). The rendered bytes are uploaded into a
+  server-created 0700 temp dir (`mktemp -d /tmp/rsmon-worker-act.XXXXXX`
+  under the SSH user) rather than a predictable `/tmp` path, eliminating
+  the symlink/TOCTOU attack surface on the upload while the token still
+  travels only as base64 over the session stdin and, later, only inside
+  the mode-0600 env file. The env is then installed with mode 0600 in the
+  config dir (mode 0750).
+- **Atomic install.** The staged binary is validated (`<stage>
+  --version`) before any state is touched, then installed with a
+  same-directory temp file + rename. The env and unit files are installed
+  the same way. The data dir (and `webapp/` subdir) is created with mode
+  0755.
+- **Supervisor.** `restart_svc` drives start/restart through whichever
+  init system is *actually running*: systemd (`systemctl`), OpenRC
+  (`rc-service`), or the embedded supervisor fallback when no init is
+  running (containers/chroots, the "no-service gate"). The health loop
+  verifies the service is active *through the supervisor* — `systemctl
+  is-active` for systemd, `rc-service status` for OpenRC, and a
+  zombie-aware pid check for the embedded supervisor — never through a
+  pid file on the init-managed paths. The embedded supervisor keeps the
+  worker as a single background process with a pid file under the data
+  dir, stops the previous instance before starting, refuses to kill a pid
+  whose executable is not the configured worker (`/proc/<pid>/exe`
+  checked before every `kill`), and answers `/healthz` via the worker's
+  own `liveness` subcommand with a clean `env -i` environment built from
+  the env file (so a rolled-back env can never leak a stale variable into
+  the restored process). The OpenRC init script exports the env-file
+  variables (`set -a` before sourcing) so the worker inherits them when
+  OpenRC starts it.
+- **Rollback.** The prior binary, env, and unit (and the unit's enable
+  state) are snapshotted into the data dir before any mutation, and an
+  `EXIT`/`HUP`/`INT`/`TERM` trap restores them (preserving the prior
+  binary/env/unit metadata via `cp -p`) and restarts the prior service if
+  any later step fails or a signal interrupts the run. A corrupt staged
+  binary fails the pre-validation *before* the trap is armed, so the
+  prior install is never touched. A fresh-install failure disables the
+  newly-enabled unit (`systemctl disable` / `rc-update del`) and removes
+  it; a rerun failure restores the prior unit and re-applies its prior
+  enable state. Rollback also removes the backup dir, the `.new` temp
+  files, the uploaded env/unit temps, and the activation lock.
+- **Lock, recovery, interruption.** A `mkdir`-based activation lock
+  (`.rsmon-activate.lock`, broken automatically when its recorded pid is
+  dead) prevents concurrent activations from racing the deployed-state
+  mutation. The snapshot is marked; a run killed mid-flight (SSH drop,
+  SIGKILL) leaves that marker, and the next activation restores the
+  leftover snapshot before proceeding, so the host never stays
+  half-activated.
+- **Idempotency.** A rerun reuses the existing env (rewriting it
+  deterministically from the same knobs), swaps the binary atomically,
+  restarts exactly one worker, and leaves no `.rsmon-backup`, `.new`,
+  `/tmp/rsmon-worker-act.*`, or lock leftovers. `--no-start` installs the
+  full layout without starting anything and reruns stay idle.
+
+The `source-install` CLI activates by default; pass `--no-activate` for
+the staging-only behavior of work package 3. The worker token is required
+for activation and is supplied with `--token`/`--token-file` or
+`--env-file` (prefer the file options; direct flags expose the value
+through the process list).
 
 ## Docker OpenSSH Test Harness
 
@@ -273,22 +354,37 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
   matches, the clone's origin is verified against the configured repository and
   then fetched in place, the resolved branch/commit record is rewritten after
   the new build succeeds, and the staging build atomically swaps over the
-  previous staging binary. The "one active service" guarantee is a work package
-  4 property (activation); work package 3 leaves no running service to
-  duplicate and no leaked toolchain temp files.
+  previous staging binary. Activation (work package 4) makes the rerun
+  idempotent at the service level too: exactly one worker process exists, the
+  env file is rewritten deterministically from the same knobs, and no
+  `.rsmon-backup`, `.new`, `/tmp/rsmon-worker-act.*`, or lock files leak.
 - Wrong host fingerprints fail before remote mutation. The harness's fresh
   known_hosts file is trust-on-first-use; its dedicated mismatch test dials
   against a known_hosts entry carrying a different server key and proves the
   dial fails before any command runs.
 - Tokens/passwords should come from files: `--key-passphrase-file`,
-  `--password-file`, and `--sudo-password-file` keep secrets out of argv and
-  shell history, while the equivalent direct flags expose them through the
-  process list. The source install itself sends no worker token or
-  control-plane credential at all, and sudo passwords travel only over the
-  session's stdin.
-- Remote temporary files are removed on success and failure.
+  `--password-file`, `--sudo-password-file`, and the activation
+  `--token-file`/`--env-file` keep secrets out of argv and shell history,
+  while the equivalent direct flags expose them through the process list. The
+  worker token is written only to the mode-0600 env file, never echoed into a
+  command or log, and sudo passwords travel only over the session's stdin. The
+  env and unit uploads land in a server-created 0700 `mktemp -d` directory (not
+  a predictable `/tmp` path), so no local user can plant a symlink at the
+  upload target (TOCTOU), and the env file itself is read/rendered exactly once
+  so a local writer cannot swap it between validation and upload.
+- Remote temporary files are removed on success and failure, including the
+  activation backup dir, the uploaded env/unit temps, and the activation lock.
+  The snapshot is written under a marker so a run killed mid-flight is
+  recovered (restored) by the next activation instead of leaving the host
+  half-activated.
 - Failed builds, failed checkouts, and failed toolchain swaps do not replace a
-  working binary or service definition.
+  working binary or service definition. Failed activations, failed starts, and
+  failed `/healthz` verifications restore the prior binary, env, and service
+  definition (preserving their metadata), re-apply the prior unit enable state
+  (or disable a freshly-enabled unit on a fresh failure), and bring the prior
+  worker back up (work package 4 rollback). The supervisor only ever kills a
+  process whose `/proc/<pid>/exe` matches the configured worker binary, so a
+  stale or recycled pid file can never kill an unrelated process.
 - Package-manager and download failures return bounded actionable errors.
 - The installer verifies Go tarball checksum before extraction.
 
@@ -300,24 +396,32 @@ RSMON_TEST_DOCKER=1 go test -v -count=1 -timeout 30m ./internal/installer/harnes
 - [x] 3. Execute source installation through the existing SSH transport
       (prerequisites, verified Go toolchain, clone/update, resolved branch and
       commit record, and a build to a staging path; no service activation).
-- [ ] 4. Add atomic build/install, idempotency, and failure rollback.
+- [x] 4. Add atomic build/install, idempotency, and failure rollback (binary,
+      env, data dir, and init service definition installed atomically; process
+      and `/healthz` verified; activation/start/health failures restore the
+      prior install; reruns keep exactly one service).
 - [ ] 5. Add Alpine, Ubuntu, and Arch network E2E tests to CI.
 - [ ] 6. Add CentOS-family support.
 - [ ] 7. Plan native Windows service and macOS launchd installers separately.
 
 ## Acceptance Gates
 
-- [ ] All three initial Linux images install from a clean state through OpenSSH.
-      Work package 3 covers the install-to-staging half (package install,
-      verified Go 1.26, clone/update, resolved commit, staging build); the
-      service-start half is the work package 4 gate.
-- [ ] The built worker reports the expected version/commit and serves `/healthz`.
-- [ ] Re-running the installer succeeds without duplicate services or leaked files.
-- [ ] Host-key, checksum, clone, build, and service-start failure tests preserve the
-      previous installation.
+- [x] All three initial Linux images install from a clean state through OpenSSH:
+      package install, verified Go 1.26, clone/update, resolved commit, staging
+      build, and (work package 4) atomic activation with a running verified
+      worker.
+- [x] The built worker reports the expected version/commit and serves `/healthz`
+      (verified by the E2E fixture tests and by the activation health gate).
+- [x] Re-running the installer succeeds without duplicate services or leaked
+      files (exactly one worker process and no backup/temp leftovers asserted by
+      the rerun test on all three fixtures).
+- [x] Host-key, checksum, clone, build, and service-start failure tests preserve
+      the previous installation (build/checkout, activation, start, and health
+      failure reruns on the Alpine fixture all restore the prior binary, env,
+      and running service).
 - [ ] CI uses approved registry mirrors and cleans every test container/network.
 
-## Verified Test Evidence (work packages 1-3)
+## Verified Test Evidence (work packages 1-4)
 
 Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
 
@@ -334,14 +438,48 @@ Recorded 2026-08-12 from `make test-ssh` (Docker Engine 29.7.1):
   prerequisite install, verified Go 1.26 toolchain download/extraction, clone
   of the public repository, resolution of its default branch, resolved-commit
   record file, and a staging build that reports the resolved commit via
-  `--version`. The public repo's `HEAD` was `master` at `4651deb2...` during
-  the run, and all three fixtures resolved to that branch and commit (the
-  installer records whatever the remote publishes). A rerun succeeds, keeps
-  the same branch, reuses the toolchain, and leaves no
-  `/tmp/rsmon-toolchain-*` temp dirs. The running service and its
-  configuration are not touched (work package 4 boundary).
-- The unit suite covers the per-step remote scripts, option validation,
-  branch/commit parsing, sudo wrapping (password never in the command), the
-  secrets-absent contract, and an in-process real-SSH orchestration flow with
-  failure paths for missing pinned branches, build failures, and detection
-  failures.
+  `--version`. A rerun succeeds, keeps the same branch, reuses the toolchain,
+  and leaves no `/tmp/rsmon-toolchain-*` temp dirs.
+- `TestSourceInstallActivationFixtures` runs the full work-package-4 flow on
+  each fixture from a clean state: after the staging build the installer
+  atomically installs the binary (`/usr/local/bin/rsmon-worker`), env
+  (`/etc/rsmon-worker/worker.env`, mode 0600; config dir 0750), data dir
+  (`/var/lib/rsmon-worker`), and the detected init's service definition
+  (systemd unit on Ubuntu/Arch, OpenRC script on Alpine), then starts the
+  worker via the embedded supervisor (no init runs inside the fixtures) and
+  verifies exactly one worker process and a live `/healthz`. A rerun succeeds,
+  keeps a single process, rewrites the env deterministically, and leaves no
+  `.rsmon-backup`, `.new`, `/tmp/rsmon-worker-act.*`, or lock leftovers.
+- `TestSourceInstallActivationFailureRollback` (Alpine) forces four rerun
+  failures and proves each restores the prior install byte-for-byte and
+  running: a dirty checkout (build failure), a sabotaged atomic swap
+  (activation failure), `WORKER_CLUSTER_ENABLED=true` without credentials (the
+  new process exits at boot; start failure), and `WORKER_HOST=255.255.255.255`
+  (the new process runs but `/healthz` is unreachable; health failure). In
+  every case the installed binary SHA-256, env, service definition, single
+  process, and liveness match the pre-failure state, and no backup, lock, or
+  temp files leak.
+- `TestSourceInstallActivationNoStart` installs the full layout without
+  starting anything, and reruns stay idle.
+- The targeted shell fixture tests (`TestActivateShell*`) execute the real
+  activation script against stub `systemctl`/`rc-service`/`rc-update`
+  implementations and cover the init-managed paths the Docker fixtures cannot
+  reach: systemd unit install/enable with a `systemctl is-active`-based health
+  loop (and no pid file), OpenRC install/enable with an `rc-service`-based
+  health loop, fresh-failure rollback disabling a newly-enabled unit, rerun
+  rollback restoring the prior unit and its enable state, stale-lock breaking,
+  and interrupted-run recovery from a leftover backup marker.
+- The unit suite covers the per-step remote scripts, the activation script
+  (backup + recovery marker, activation lock, atomic install, rollback trap,
+  supervisor, pid/zombie handling, pid-belongs-to-binary checks, enable-state
+  restoration, secrets absent), OpenRC unit rendering (including env
+  export), option validation, branch/commit parsing, sudo wrapping (password
+  never in the command), and an in-process real-SSH orchestration flow with
+  failure paths for missing pinned branches, build failures, detection
+  failures, and activation/start/health failures.
+- **Residual limitation (stated honestly):** live systemd and OpenRC cannot
+  run inside the Docker/OpenSSH harness (the container PID 1 is sshd), so the
+  init-managed supervisor paths are verified by the stub-based shell fixture
+  tests above rather than against a real init. The stub tools simulate unit
+  state and command flow, not real systemd/OpenRC unit semantics; a real
+  init-system smoke test on a booted host remains a follow-up.