# Rollback and no-model-recovery acceptance This is the execution guide for the **rollback** (`updates-and-rollback`) and **no-model recovery** (`no-model-recovery`) release gates. It describes how to run the acceptance journey in the disposable VM harness, what it asserts, where the evidence lands, and how a failure is diagnosed. > **Status: NOT YET RUN.** The harness and its offline tests exist and pass on a > plain host, but the live journey has **not** been executed. Until a real run > succeeds and its `results.json` is imported, both release gates stay > `blocked`. A run made with a fake transport is marked `mocked` and is > **rejected** by the release tool. ## What the journey proves The installer writes a coordinated `@` + `@var` known-good snapshot at install time. The journey then deliberately changes the disk and asks the real `sys undo` to put it back. The ordered steps (from `installer/tests/vm/rollback_acceptance.py`) are: 1. `sys-binary-identity` — the installed `sys` matches the pinned digest. 2. `known-good-coordinated-snapshot` — a committed root snapshot with a paired `@var` snapshot exists. 3. `home-data-marker` — a marker is written into `/home` (a separate subvolume). 4. `controlled-package-change` — a small package is installed, adding a `/var/db/pkg` entry and files under `/`. 5. `deterministic-guest-breakage` — `/etc/hostname` is overwritten with a sentinel. 6. `sys-undo-activates` — `sys undo ` writes an `activated` rollback state for the next boot. 7. `reboot-detached` — the installed disk boots with the installer ISO absent. 8. `restored-root-var-generations` — the booted `rootflags`, `/etc/fstab` `/` and `/var` subvolumes match the rollback state. 9. `package-files-restored` — the changed package's DB entry and files are gone, and the sentinel hostname is reverted. 10. `portage-database-consistent` — `/var/db/pkg` equals its pre-change listing and every entry has a `CONTENTS` file. 11. `kernel-modules-compatible` — the running kernel has a module tree in the restored root. 12. `home-data-survives` — the `/home` marker is byte-identical. 13. `recovery-entry-armed` — `bootctl set-oneshot phoebus-recovery` succeeds. 14. `recovery-boot-no-network-no-model` — the recovery entry boots `rescue.target`, no non-loopback interface is up, `sys` status works offline, and `sys ask` reports the deterministic no-model fallback. The package changed in step 4 is controlled by `ROLLBACK_PACKAGE` (default `app-misc/hello`); pick a small, signed binary package for a fast run. ## Prerequisites - A **disposable qcow2** already installed by an existing harness run (`serial-driver.py install` or `appliance-driver.py install`). This driver never installs and never attaches the installer ISO. - `/var/tmp/phoebus-vm/disks` (or `PHOEBUS_VM_DISKS`) has the disk. Never point a writable disk at `/dev/*`; the harness refuses it. - `qemu-system-x86_64`, `qemu-img`, OVMF firmware, Python `pexpect`. - The candidate ISO and the `sys` binary under test, as files on the host, so their digests can be pinned. - Network access during the change phase (the package install); the recovery phase uses `-net none`. ## Running it Print the plan without starting a VM (safe on any host): ```sh python3 installer/tests/vm/rollback-driver.py plan ``` Run the full journey: ```sh PHOEBUS_VM_DISKS=/var/tmp/phoebus-vm/disks \ DISK=/var/tmp/phoebus-vm/disks/rollback.qcow2 \ OVMF_CODE=/usr/share/OVMF/OVMF_CODE_4M.fd \ OVMF_VARS=/usr/share/OVMF/OVMF_VARS_4M.fd \ ISO=/var/tmp/phoebus-vm/phoebus-installer-amd64.iso \ PHOEBUS_VM_SYS=/var/tmp/phoebus-vm/sys \ PHOEBUS_VM_ARTIFACT=phoebus-installer-amd64.iso \ PHOEBUS_VM_ARTIFACT_SHA256="$(sha256sum /var/tmp/phoebus-vm/phoebus-installer-amd64.iso | cut -d' ' -f1)" \ PHOEBUS_VM_STRICT=1 \ LUKS_PASSPHRASE=phoebus-luks-test USER_PASSWORD=phoebus-user-test \ python3 installer/tests/vm/rollback-driver.py run ``` Re-verify only the recovery boot (boots normally once to arm the one-shot, then reboots with no network): ```sh # same environment, then: python3 installer/tests/vm/rollback-driver.py recovery ``` `PHOEBUS_VM_ARTIFACT[_SHA256]` is what binds the evidence to a candidate; the release tool refuses unbound or stale evidence. `PHOEBUS_VM_SYS` is hashed and pinned as the `sys` identity (override with `PHOEBUS_VM_SYS_SHA256`). The guest must report the same digest in step 1. ## Artifacts Under `$PHOEBUS_VM_STATE//` (default `/var/tmp/phoebus-vm/`): - `results.json` — the `rollback-acceptance` scenario. - `recovery/results.json` — the `rollback-recovery` scenario (written by `run`). - `serial-boot1.log`, `serial-boot2.log`, `serial-boot3.log` — per-boot serial transcripts. - `OVMF_VARS.fd` — the run's own writable firmware variables. `results.json` records `scenario`, `run_id`, `evidence` (`live`), `exit_code`, `counts`, `artifact` (name + ISO digest), `inputs` (`iso` and `sys` digests) and the per-check `checks` list. ## Feeding the release gates Import the run directory when producing a gate report: ```sh installer-rs/target/release/phoebus-release gates --execute \ --candidate /path/phoebus-installer-amd64.iso \ --harness-results /var/tmp/phoebus-vm/ \ --output gate-report.json ``` The `updates-and-rollback` gate resolves `rollback-acceptance`; the `no-model-recovery` gate resolves `rollback-recovery`. A gate **fails** (never passes) when the evidence is: - `mocked` (a fake-transport unit run), - missing the `iso` or `sys` pinned input, - bound to a different candidate digest (stale), - unbound while a candidate is supplied, or - present but with `exit_code != 0`. ## Failure diagnosis | Symptom | Likely cause | |---|---| | `sys-binary-identity` fails | the installed `sys` is not the build under test; rebuild the ISO from the merged branch and re-install | | `known-good-coordinated-snapshot` fails | the install did not record a paired `@var` snapshot, or `snapshot-history.toml` is unreadable (it is never treated as empty) | | `controlled-package-change` fails/times out | no network, no signed binhost, or the atom needs a long compile; choose another `ROLLBACK_PACKAGE` | | `sys-undo-activates` fails | `sys undo` refused (no committed snapshot, unreadable history) or the broker exited nonzero | | `restored-root-var-generations` fails | `rootflags`/fstab were not rewritten to the restored generation | | `portage-database-consistent` fails | root and `/var` came from different generations | | `kernel-modules-compatible` fails | the selected boot entry does not match the restored `lib/modules` | | `recovery-no-network` fails | QEMU was still given a network device; check `-net none` | | `recovery-no-model-fallback` fails | a provider is configured, or `sys ask` did not fail closed offline | | harness exits 2 with `disposable-disk` | the disk is not a disposable qcow2 inside `PHOEBUS_VM_DISKS` | ## Integration note The branch this work is built on (`codex/rollback-acceptance`) is based on the CI-hardening commit `1d37fbc` and does **not** include the sibling `sys` hardening commit `4679f93`. The installed `sys` used in a live run must come from a build that contains the sys hardening; otherwise step 1 (`sys-binary-identity`) will correctly fail. Build the acceptance ISO after the two branches are integrated.