156 lines
7.2 KiB
Markdown
156 lines
7.2 KiB
Markdown
# Rollback and no-model-recovery acceptance
|
|
|
|
This is the execution guide for the **rollback** (`updates-and-rollback`) and
|
|
**no-model recovery** (`no-model-recovery`) release gates. It describes how to
|
|
run the acceptance journey in the disposable VM harness, what it asserts, where
|
|
the evidence lands, and how a failure is diagnosed.
|
|
|
|
> **Status: NOT YET RUN.** The harness and its offline tests exist and pass on a
|
|
> plain host, but the live journey has **not** been executed. Until a real run
|
|
> succeeds and its `results.json` is imported, both release gates stay
|
|
> `blocked`. A run made with a fake transport is marked `mocked` and is
|
|
> **rejected** by the release tool.
|
|
|
|
## What the journey proves
|
|
|
|
The installer writes a coordinated `@` + `@var` known-good snapshot at install
|
|
time. The journey then deliberately changes the disk and asks the real
|
|
`sys undo` to put it back. The ordered steps (from
|
|
`installer/tests/vm/rollback_acceptance.py`) are:
|
|
|
|
1. `sys-binary-identity` — the installed `sys` matches the pinned digest.
|
|
2. `known-good-coordinated-snapshot` — a committed root snapshot with a paired
|
|
`@var` snapshot exists.
|
|
3. `home-data-marker` — a marker is written into `/home` (a separate subvolume).
|
|
4. `controlled-package-change` — a small package is installed, adding a
|
|
`/var/db/pkg` entry and files under `/`.
|
|
5. `deterministic-guest-breakage` — `/etc/hostname` is overwritten with a
|
|
sentinel.
|
|
6. `sys-undo-activates` — `sys undo <id>` writes an `activated` rollback state
|
|
for the next boot.
|
|
7. `reboot-detached` — the installed disk boots with the installer ISO absent.
|
|
8. `restored-root-var-generations` — the booted `rootflags`, `/etc/fstab` `/`
|
|
and `/var` subvolumes match the rollback state.
|
|
9. `package-files-restored` — the changed package's DB entry and files are gone,
|
|
and the sentinel hostname is reverted.
|
|
10. `portage-database-consistent` — `/var/db/pkg` equals its pre-change listing
|
|
and every entry has a `CONTENTS` file.
|
|
11. `kernel-modules-compatible` — the running kernel has a module tree in the
|
|
restored root.
|
|
12. `home-data-survives` — the `/home` marker is byte-identical.
|
|
13. `recovery-entry-armed` — `bootctl set-oneshot phoebus-recovery` succeeds.
|
|
14. `recovery-boot-no-network-no-model` — the recovery entry boots
|
|
`rescue.target`, no non-loopback interface is up, `sys` status works
|
|
offline, and `sys ask` reports the deterministic no-model fallback.
|
|
|
|
The package changed in step 4 is controlled by `ROLLBACK_PACKAGE` (default
|
|
`app-misc/hello`); pick a small, signed binary package for a fast run.
|
|
|
|
## Prerequisites
|
|
|
|
- A **disposable qcow2** already installed by an existing harness run
|
|
(`serial-driver.py install` or `appliance-driver.py install`). This driver
|
|
never installs and never attaches the installer ISO.
|
|
- `/var/tmp/phoebus-vm/disks` (or `PHOEBUS_VM_DISKS`) has the disk. Never point
|
|
a writable disk at `/dev/*`; the harness refuses it.
|
|
- `qemu-system-x86_64`, `qemu-img`, OVMF firmware, Python `pexpect`.
|
|
- The candidate ISO and the `sys` binary under test, as files on the host, so
|
|
their digests can be pinned.
|
|
- Network access during the change phase (the package install); the recovery
|
|
phase uses `-net none`.
|
|
|
|
## Running it
|
|
|
|
Print the plan without starting a VM (safe on any host):
|
|
|
|
```sh
|
|
python3 installer/tests/vm/rollback-driver.py plan
|
|
```
|
|
|
|
Run the full journey:
|
|
|
|
```sh
|
|
PHOEBUS_VM_DISKS=/var/tmp/phoebus-vm/disks \
|
|
DISK=/var/tmp/phoebus-vm/disks/rollback.qcow2 \
|
|
OVMF_CODE=/usr/share/OVMF/OVMF_CODE_4M.fd \
|
|
OVMF_VARS=/usr/share/OVMF/OVMF_VARS_4M.fd \
|
|
ISO=/var/tmp/phoebus-vm/phoebus-installer-amd64.iso \
|
|
PHOEBUS_VM_SYS=/var/tmp/phoebus-vm/sys \
|
|
PHOEBUS_VM_ARTIFACT=phoebus-installer-amd64.iso \
|
|
PHOEBUS_VM_ARTIFACT_SHA256="$(sha256sum /var/tmp/phoebus-vm/phoebus-installer-amd64.iso | cut -d' ' -f1)" \
|
|
PHOEBUS_VM_STRICT=1 \
|
|
LUKS_PASSPHRASE=phoebus-luks-test USER_PASSWORD=phoebus-user-test \
|
|
python3 installer/tests/vm/rollback-driver.py run
|
|
```
|
|
|
|
Re-verify only the recovery boot (boots normally once to arm the one-shot, then
|
|
reboots with no network):
|
|
|
|
```sh
|
|
# same environment, then:
|
|
python3 installer/tests/vm/rollback-driver.py recovery
|
|
```
|
|
|
|
`PHOEBUS_VM_ARTIFACT[_SHA256]` is what binds the evidence to a candidate; the
|
|
release tool refuses unbound or stale evidence. `PHOEBUS_VM_SYS` is hashed and
|
|
pinned as the `sys` identity (override with `PHOEBUS_VM_SYS_SHA256`). The guest
|
|
must report the same digest in step 1.
|
|
|
|
## Artifacts
|
|
|
|
Under `$PHOEBUS_VM_STATE/<run-id>/` (default `/var/tmp/phoebus-vm/`):
|
|
|
|
- `results.json` — the `rollback-acceptance` scenario.
|
|
- `recovery/results.json` — the `rollback-recovery` scenario (written by `run`).
|
|
- `serial-boot1.log`, `serial-boot2.log`, `serial-boot3.log` — per-boot serial
|
|
transcripts.
|
|
- `OVMF_VARS.fd` — the run's own writable firmware variables.
|
|
|
|
`results.json` records `scenario`, `run_id`, `evidence` (`live`), `exit_code`,
|
|
`counts`, `artifact` (name + ISO digest), `inputs` (`iso` and `sys` digests) and
|
|
the per-check `checks` list.
|
|
|
|
## Feeding the release gates
|
|
|
|
Import the run directory when producing a gate report:
|
|
|
|
```sh
|
|
installer-rs/target/release/phoebus-release gates --execute \
|
|
--candidate /path/phoebus-installer-amd64.iso \
|
|
--harness-results /var/tmp/phoebus-vm/<run-id> \
|
|
--output gate-report.json
|
|
```
|
|
|
|
The `updates-and-rollback` gate resolves `rollback-acceptance`; the
|
|
`no-model-recovery` gate resolves `rollback-recovery`. A gate **fails**
|
|
(never passes) when the evidence is:
|
|
|
|
- `mocked` (a fake-transport unit run),
|
|
- missing the `iso` or `sys` pinned input,
|
|
- bound to a different candidate digest (stale),
|
|
- unbound while a candidate is supplied, or
|
|
- present but with `exit_code != 0`.
|
|
|
|
## Failure diagnosis
|
|
|
|
| Symptom | Likely cause |
|
|
|---|---|
|
|
| `sys-binary-identity` fails | the installed `sys` is not the build under test; rebuild the ISO from the merged branch and re-install |
|
|
| `known-good-coordinated-snapshot` fails | the install did not record a paired `@var` snapshot, or `snapshot-history.toml` is unreadable (it is never treated as empty) |
|
|
| `controlled-package-change` fails/times out | no network, no signed binhost, or the atom needs a long compile; choose another `ROLLBACK_PACKAGE` |
|
|
| `sys-undo-activates` fails | `sys undo` refused (no committed snapshot, unreadable history) or the broker exited nonzero |
|
|
| `restored-root-var-generations` fails | `rootflags`/fstab were not rewritten to the restored generation |
|
|
| `portage-database-consistent` fails | root and `/var` came from different generations |
|
|
| `kernel-modules-compatible` fails | the selected boot entry does not match the restored `lib/modules` |
|
|
| `recovery-no-network` fails | QEMU was still given a network device; check `-net none` |
|
|
| `recovery-no-model-fallback` fails | a provider is configured, or `sys ask` did not fail closed offline |
|
|
| harness exits 2 with `disposable-disk` | the disk is not a disposable qcow2 inside `PHOEBUS_VM_DISKS` |
|
|
|
|
## Integration note
|
|
|
|
The branch this work is built on (`codex/rollback-acceptance`) is based on the
|
|
CI-hardening commit `1d37fbc` and does **not** include the sibling `sys`
|
|
hardening commit `4679f93`. The installed `sys` used in a live run must come
|
|
from a build that contains the sys hardening; otherwise step 1
|
|
(`sys-binary-identity`) will correctly fail. Build the acceptance ISO after the
|
|
two branches are integrated.
|