phoebus_os/docs/ROLLBACK-ACCEPTANCE.md

7.2 KiB

Rollback and no-model-recovery acceptance

This is the execution guide for the rollback (updates-and-rollback) and no-model recovery (no-model-recovery) release gates. It describes how to run the acceptance journey in the disposable VM harness, what it asserts, where the evidence lands, and how a failure is diagnosed.

Status: NOT YET RUN. The harness and its offline tests exist and pass on a plain host, but the live journey has not been executed. Until a real run succeeds and its results.json is imported, both release gates stay blocked. A run made with a fake transport is marked mocked and is rejected by the release tool.

What the journey proves

The installer writes a coordinated @ + @var known-good snapshot at install time. The journey then deliberately changes the disk and asks the real sys undo to put it back. The ordered steps (from installer/tests/vm/rollback_acceptance.py) are:

  1. sys-binary-identity — the installed sys matches the pinned digest.
  2. known-good-coordinated-snapshot — a committed root snapshot with a paired @var snapshot exists.
  3. home-data-marker — a marker is written into /home (a separate subvolume).
  4. controlled-package-change — a small package is installed, adding a /var/db/pkg entry and files under /.
  5. deterministic-guest-breakage — /etc/hostname is overwritten with a sentinel.
  6. sys-undo-activates — sys undo <id> writes an activated rollback state for the next boot.
  7. reboot-detached — the installed disk boots with the installer ISO absent.
  8. restored-root-var-generations — the booted rootflags, /etc/fstab / and /var subvolumes match the rollback state.
  9. package-files-restored — the changed package's DB entry and files are gone, and the sentinel hostname is reverted.
  10. portage-database-consistent — /var/db/pkg equals its pre-change listing and every entry has a CONTENTS file.
  11. kernel-modules-compatible — the running kernel has a module tree in the restored root.
  12. home-data-survives — the /home marker is byte-identical.
  13. recovery-entry-armed — bootctl set-oneshot phoebus-recovery succeeds.
  14. recovery-boot-no-network-no-model — the recovery entry boots rescue.target, no non-loopback interface is up, sys status works offline, and sys ask reports the deterministic no-model fallback.

The package changed in step 4 is controlled by ROLLBACK_PACKAGE (default app-misc/hello); pick a small, signed binary package for a fast run.

Prerequisites

  • A disposable qcow2 already installed by an existing harness run (serial-driver.py install or appliance-driver.py install). This driver never installs and never attaches the installer ISO.
  • /var/tmp/phoebus-vm/disks (or PHOEBUS_VM_DISKS) has the disk. Never point a writable disk at /dev/*; the harness refuses it.
  • qemu-system-x86_64, qemu-img, OVMF firmware, Python pexpect.
  • The candidate ISO and the sys binary under test, as files on the host, so their digests can be pinned.
  • Network access during the change phase (the package install); the recovery phase uses -net none.

Running it

Print the plan without starting a VM (safe on any host):

python3 installer/tests/vm/rollback-driver.py plan

Run the full journey:

PHOEBUS_VM_DISKS=/var/tmp/phoebus-vm/disks \
DISK=/var/tmp/phoebus-vm/disks/rollback.qcow2 \
OVMF_CODE=/usr/share/OVMF/OVMF_CODE_4M.fd \
OVMF_VARS=/usr/share/OVMF/OVMF_VARS_4M.fd \
ISO=/var/tmp/phoebus-vm/phoebus-installer-amd64.iso \
PHOEBUS_VM_SYS=/var/tmp/phoebus-vm/sys \
PHOEBUS_VM_ARTIFACT=phoebus-installer-amd64.iso \
PHOEBUS_VM_ARTIFACT_SHA256="$(sha256sum /var/tmp/phoebus-vm/phoebus-installer-amd64.iso | cut -d' ' -f1)" \
PHOEBUS_VM_STRICT=1 \
LUKS_PASSPHRASE=phoebus-luks-test USER_PASSWORD=phoebus-user-test \
python3 installer/tests/vm/rollback-driver.py run

Re-verify only the recovery boot (boots normally once to arm the one-shot, then reboots with no network):

# same environment, then:
python3 installer/tests/vm/rollback-driver.py recovery

PHOEBUS_VM_ARTIFACT[_SHA256] is what binds the evidence to a candidate; the release tool refuses unbound or stale evidence. PHOEBUS_VM_SYS is hashed and pinned as the sys identity (override with PHOEBUS_VM_SYS_SHA256). The guest must report the same digest in step 1.

Artifacts

Under $PHOEBUS_VM_STATE/<run-id>/ (default /var/tmp/phoebus-vm/):

  • results.json — the rollback-acceptance scenario.
  • recovery/results.json — the rollback-recovery scenario (written by run).
  • serial-boot1.log, serial-boot2.log, serial-boot3.log — per-boot serial transcripts.
  • OVMF_VARS.fd — the run's own writable firmware variables.

results.json records scenario, run_id, evidence (live), exit_code, counts, artifact (name + ISO digest), inputs (iso and sys digests) and the per-check checks list.

Feeding the release gates

Import the run directory when producing a gate report:

installer-rs/target/release/phoebus-release gates --execute \
  --candidate /path/phoebus-installer-amd64.iso \
  --harness-results /var/tmp/phoebus-vm/<run-id> \
  --output gate-report.json

The updates-and-rollback gate resolves rollback-acceptance; the no-model-recovery gate resolves rollback-recovery. A gate fails (never passes) when the evidence is:

  • mocked (a fake-transport unit run),
  • missing the iso or sys pinned input,
  • bound to a different candidate digest (stale),
  • unbound while a candidate is supplied, or
  • present but with exit_code != 0.

Failure diagnosis

Symptom Likely cause
sys-binary-identity fails the installed sys is not the build under test; rebuild the ISO from the merged branch and re-install
known-good-coordinated-snapshot fails the install did not record a paired @var snapshot, or snapshot-history.toml is unreadable (it is never treated as empty)
controlled-package-change fails/times out no network, no signed binhost, or the atom needs a long compile; choose another ROLLBACK_PACKAGE
sys-undo-activates fails sys undo refused (no committed snapshot, unreadable history) or the broker exited nonzero
restored-root-var-generations fails rootflags/fstab were not rewritten to the restored generation
portage-database-consistent fails root and /var came from different generations
kernel-modules-compatible fails the selected boot entry does not match the restored lib/modules
recovery-no-network fails QEMU was still given a network device; check -net none
recovery-no-model-fallback fails a provider is configured, or sys ask did not fail closed offline
harness exits 2 with disposable-disk the disk is not a disposable qcow2 inside PHOEBUS_VM_DISKS

Integration note

The branch this work is built on (codex/rollback-acceptance) is based on the CI-hardening commit 1d37fbc and does not include the sibling sys hardening commit 4679f93. The installed sys used in a live run must come from a build that contains the sys hardening; otherwise step 1 (sys-binary-identity) will correctly fail. Build the acceptance ISO after the two branches are integrated.