phoebus_os/docs/ROLLBACK-ACCEPTANCE.md

156 lines
7.2 KiB
Markdown

# Rollback and no-model-recovery acceptance
This is the execution guide for the **rollback** (`updates-and-rollback`) and
**no-model recovery** (`no-model-recovery`) release gates. It describes how to
run the acceptance journey in the disposable VM harness, what it asserts, where
the evidence lands, and how a failure is diagnosed.
> **Status: NOT YET RUN.** The harness and its offline tests exist and pass on a
> plain host, but the live journey has **not** been executed. Until a real run
> succeeds and its `results.json` is imported, both release gates stay
> `blocked`. A run made with a fake transport is marked `mocked` and is
> **rejected** by the release tool.
## What the journey proves
The installer writes a coordinated `@` + `@var` known-good snapshot at install
time. The journey then deliberately changes the disk and asks the real
`sys undo` to put it back. The ordered steps (from
`installer/tests/vm/rollback_acceptance.py`) are:
1. `sys-binary-identity` — the installed `sys` matches the pinned digest.
2. `known-good-coordinated-snapshot` — a committed root snapshot with a paired
`@var` snapshot exists.
3. `home-data-marker` — a marker is written into `/home` (a separate subvolume).
4. `controlled-package-change` — a small package is installed, adding a
`/var/db/pkg` entry and files under `/`.
5. `deterministic-guest-breakage` — `/etc/hostname` is overwritten with a
sentinel.
6. `sys-undo-activates` — `sys undo <id>` writes an `activated` rollback state
for the next boot.
7. `reboot-detached` — the installed disk boots with the installer ISO absent.
8. `restored-root-var-generations` — the booted `rootflags`, `/etc/fstab` `/`
and `/var` subvolumes match the rollback state.
9. `package-files-restored` — the changed package's DB entry and files are gone,
and the sentinel hostname is reverted.
10. `portage-database-consistent` — `/var/db/pkg` equals its pre-change listing
and every entry has a `CONTENTS` file.
11. `kernel-modules-compatible` — the running kernel has a module tree in the
restored root.
12. `home-data-survives` — the `/home` marker is byte-identical.
13. `recovery-entry-armed` — `bootctl set-oneshot phoebus-recovery` succeeds.
14. `recovery-boot-no-network-no-model` — the recovery entry boots
`rescue.target`, no non-loopback interface is up, `sys` status works
offline, and `sys ask` reports the deterministic no-model fallback.
The package changed in step 4 is controlled by `ROLLBACK_PACKAGE` (default
`app-misc/hello`); pick a small, signed binary package for a fast run.
## Prerequisites
- A **disposable qcow2** already installed by an existing harness run
(`serial-driver.py install` or `appliance-driver.py install`). This driver
never installs and never attaches the installer ISO.
- `/var/tmp/phoebus-vm/disks` (or `PHOEBUS_VM_DISKS`) has the disk. Never point
a writable disk at `/dev/*`; the harness refuses it.
- `qemu-system-x86_64`, `qemu-img`, OVMF firmware, Python `pexpect`.
- The candidate ISO and the `sys` binary under test, as files on the host, so
their digests can be pinned.
- Network access during the change phase (the package install); the recovery
phase uses `-net none`.
## Running it
Print the plan without starting a VM (safe on any host):
```sh
python3 installer/tests/vm/rollback-driver.py plan
```
Run the full journey:
```sh
PHOEBUS_VM_DISKS=/var/tmp/phoebus-vm/disks \
DISK=/var/tmp/phoebus-vm/disks/rollback.qcow2 \
OVMF_CODE=/usr/share/OVMF/OVMF_CODE_4M.fd \
OVMF_VARS=/usr/share/OVMF/OVMF_VARS_4M.fd \
ISO=/var/tmp/phoebus-vm/phoebus-installer-amd64.iso \
PHOEBUS_VM_SYS=/var/tmp/phoebus-vm/sys \
PHOEBUS_VM_ARTIFACT=phoebus-installer-amd64.iso \
PHOEBUS_VM_ARTIFACT_SHA256="$(sha256sum /var/tmp/phoebus-vm/phoebus-installer-amd64.iso | cut -d' ' -f1)" \
PHOEBUS_VM_STRICT=1 \
LUKS_PASSPHRASE=phoebus-luks-test USER_PASSWORD=phoebus-user-test \
python3 installer/tests/vm/rollback-driver.py run
```
Re-verify only the recovery boot (boots normally once to arm the one-shot, then
reboots with no network):
```sh
# same environment, then:
python3 installer/tests/vm/rollback-driver.py recovery
```
`PHOEBUS_VM_ARTIFACT[_SHA256]` is what binds the evidence to a candidate; the
release tool refuses unbound or stale evidence. `PHOEBUS_VM_SYS` is hashed and
pinned as the `sys` identity (override with `PHOEBUS_VM_SYS_SHA256`). The guest
must report the same digest in step 1.
## Artifacts
Under `$PHOEBUS_VM_STATE/<run-id>/` (default `/var/tmp/phoebus-vm/`):
- `results.json` — the `rollback-acceptance` scenario.
- `recovery/results.json` — the `rollback-recovery` scenario (written by `run`).
- `serial-boot1.log`, `serial-boot2.log`, `serial-boot3.log` — per-boot serial
transcripts.
- `OVMF_VARS.fd` — the run's own writable firmware variables.
`results.json` records `scenario`, `run_id`, `evidence` (`live`), `exit_code`,
`counts`, `artifact` (name + ISO digest), `inputs` (`iso` and `sys` digests) and
the per-check `checks` list.
## Feeding the release gates
Import the run directory when producing a gate report:
```sh
installer-rs/target/release/phoebus-release gates --execute \
--candidate /path/phoebus-installer-amd64.iso \
--harness-results /var/tmp/phoebus-vm/<run-id> \
--output gate-report.json
```
The `updates-and-rollback` gate resolves `rollback-acceptance`; the
`no-model-recovery` gate resolves `rollback-recovery`. A gate **fails**
(never passes) when the evidence is:
- `mocked` (a fake-transport unit run),
- missing the `iso` or `sys` pinned input,
- bound to a different candidate digest (stale),
- unbound while a candidate is supplied, or
- present but with `exit_code != 0`.
## Failure diagnosis
| Symptom | Likely cause |
|---|---|
| `sys-binary-identity` fails | the installed `sys` is not the build under test; rebuild the ISO from the merged branch and re-install |
| `known-good-coordinated-snapshot` fails | the install did not record a paired `@var` snapshot, or `snapshot-history.toml` is unreadable (it is never treated as empty) |
| `controlled-package-change` fails/times out | no network, no signed binhost, or the atom needs a long compile; choose another `ROLLBACK_PACKAGE` |
| `sys-undo-activates` fails | `sys undo` refused (no committed snapshot, unreadable history) or the broker exited nonzero |
| `restored-root-var-generations` fails | `rootflags`/fstab were not rewritten to the restored generation |
| `portage-database-consistent` fails | root and `/var` came from different generations |
| `kernel-modules-compatible` fails | the selected boot entry does not match the restored `lib/modules` |
| `recovery-no-network` fails | QEMU was still given a network device; check `-net none` |
| `recovery-no-model-fallback` fails | a provider is configured, or `sys ask` did not fail closed offline |
| harness exits 2 with `disposable-disk` | the disk is not a disposable qcow2 inside `PHOEBUS_VM_DISKS` |
## Integration note
The branch this work is built on (`codex/rollback-acceptance`) is based on the
CI-hardening commit `1d37fbc` and does **not** include the sibling `sys`
hardening commit `4679f93`. The installed `sys` used in a live run must come
from a build that contains the sys hardening; otherwise step 1
(`sys-binary-identity`) will correctly fail. Build the acceptance ISO after the
two branches are integrated.