7.2 KiB
Rollback and no-model-recovery acceptance
This is the execution guide for the rollback (updates-and-rollback) and
no-model recovery (no-model-recovery) release gates. It describes how to
run the acceptance journey in the disposable VM harness, what it asserts, where
the evidence lands, and how a failure is diagnosed.
Status: NOT YET RUN. The harness and its offline tests exist and pass on a plain host, but the live journey has not been executed. Until a real run succeeds and its
results.jsonis imported, both release gates stayblocked. A run made with a fake transport is markedmockedand is rejected by the release tool.
What the journey proves
The installer writes a coordinated @ + @var known-good snapshot at install
time. The journey then deliberately changes the disk and asks the real
sys undo to put it back. The ordered steps (from
installer/tests/vm/rollback_acceptance.py) are:
sys-binary-identity— the installedsysmatches the pinned digest.known-good-coordinated-snapshot— a committed root snapshot with a paired@varsnapshot exists.home-data-marker— a marker is written into/home(a separate subvolume).controlled-package-change— a small package is installed, adding a/var/db/pkgentry and files under/.deterministic-guest-breakage—/etc/hostnameis overwritten with a sentinel.sys-undo-activates—sys undo <id>writes anactivatedrollback state for the next boot.reboot-detached— the installed disk boots with the installer ISO absent.restored-root-var-generations— the bootedrootflags,/etc/fstab/and/varsubvolumes match the rollback state.package-files-restored— the changed package's DB entry and files are gone, and the sentinel hostname is reverted.portage-database-consistent—/var/db/pkgequals its pre-change listing and every entry has aCONTENTSfile.kernel-modules-compatible— the running kernel has a module tree in the restored root.home-data-survives— the/homemarker is byte-identical.recovery-entry-armed—bootctl set-oneshot phoebus-recoverysucceeds.recovery-boot-no-network-no-model— the recovery entry bootsrescue.target, no non-loopback interface is up,sysstatus works offline, andsys askreports the deterministic no-model fallback.
The package changed in step 4 is controlled by ROLLBACK_PACKAGE (default
app-misc/hello); pick a small, signed binary package for a fast run.
Prerequisites
- A disposable qcow2 already installed by an existing harness run
(
serial-driver.py installorappliance-driver.py install). This driver never installs and never attaches the installer ISO. /var/tmp/phoebus-vm/disks(orPHOEBUS_VM_DISKS) has the disk. Never point a writable disk at/dev/*; the harness refuses it.qemu-system-x86_64,qemu-img, OVMF firmware, Pythonpexpect.- The candidate ISO and the
sysbinary under test, as files on the host, so their digests can be pinned. - Network access during the change phase (the package install); the recovery
phase uses
-net none.
Running it
Print the plan without starting a VM (safe on any host):
python3 installer/tests/vm/rollback-driver.py plan
Run the full journey:
PHOEBUS_VM_DISKS=/var/tmp/phoebus-vm/disks \
DISK=/var/tmp/phoebus-vm/disks/rollback.qcow2 \
OVMF_CODE=/usr/share/OVMF/OVMF_CODE_4M.fd \
OVMF_VARS=/usr/share/OVMF/OVMF_VARS_4M.fd \
ISO=/var/tmp/phoebus-vm/phoebus-installer-amd64.iso \
PHOEBUS_VM_SYS=/var/tmp/phoebus-vm/sys \
PHOEBUS_VM_ARTIFACT=phoebus-installer-amd64.iso \
PHOEBUS_VM_ARTIFACT_SHA256="$(sha256sum /var/tmp/phoebus-vm/phoebus-installer-amd64.iso | cut -d' ' -f1)" \
PHOEBUS_VM_STRICT=1 \
LUKS_PASSPHRASE=phoebus-luks-test USER_PASSWORD=phoebus-user-test \
python3 installer/tests/vm/rollback-driver.py run
Re-verify only the recovery boot (boots normally once to arm the one-shot, then reboots with no network):
# same environment, then:
python3 installer/tests/vm/rollback-driver.py recovery
PHOEBUS_VM_ARTIFACT[_SHA256] is what binds the evidence to a candidate; the
release tool refuses unbound or stale evidence. PHOEBUS_VM_SYS is hashed and
pinned as the sys identity (override with PHOEBUS_VM_SYS_SHA256). The guest
must report the same digest in step 1.
Artifacts
Under $PHOEBUS_VM_STATE/<run-id>/ (default /var/tmp/phoebus-vm/):
results.json— therollback-acceptancescenario.recovery/results.json— therollback-recoveryscenario (written byrun).serial-boot1.log,serial-boot2.log,serial-boot3.log— per-boot serial transcripts.OVMF_VARS.fd— the run's own writable firmware variables.
results.json records scenario, run_id, evidence (live), exit_code,
counts, artifact (name + ISO digest), inputs (iso and sys digests) and
the per-check checks list.
Feeding the release gates
Import the run directory when producing a gate report:
installer-rs/target/release/phoebus-release gates --execute \
--candidate /path/phoebus-installer-amd64.iso \
--harness-results /var/tmp/phoebus-vm/<run-id> \
--output gate-report.json
The updates-and-rollback gate resolves rollback-acceptance; the
no-model-recovery gate resolves rollback-recovery. A gate fails
(never passes) when the evidence is:
mocked(a fake-transport unit run),- missing the
isoorsyspinned input, - bound to a different candidate digest (stale),
- unbound while a candidate is supplied, or
- present but with
exit_code != 0.
Failure diagnosis
| Symptom | Likely cause |
|---|---|
sys-binary-identity fails |
the installed sys is not the build under test; rebuild the ISO from the merged branch and re-install |
known-good-coordinated-snapshot fails |
the install did not record a paired @var snapshot, or snapshot-history.toml is unreadable (it is never treated as empty) |
controlled-package-change fails/times out |
no network, no signed binhost, or the atom needs a long compile; choose another ROLLBACK_PACKAGE |
sys-undo-activates fails |
sys undo refused (no committed snapshot, unreadable history) or the broker exited nonzero |
restored-root-var-generations fails |
rootflags/fstab were not rewritten to the restored generation |
portage-database-consistent fails |
root and /var came from different generations |
kernel-modules-compatible fails |
the selected boot entry does not match the restored lib/modules |
recovery-no-network fails |
QEMU was still given a network device; check -net none |
recovery-no-model-fallback fails |
a provider is configured, or sys ask did not fail closed offline |
harness exits 2 with disposable-disk |
the disk is not a disposable qcow2 inside PHOEBUS_VM_DISKS |
Integration note
The branch this work is built on (codex/rollback-acceptance) is based on the
CI-hardening commit 1d37fbc and does not include the sibling sys
hardening commit 4679f93. The installed sys used in a live run must come
from a build that contains the sys hardening; otherwise step 1
(sys-binary-identity) will correctly fail. Build the acceptance ISO after the
two branches are integrated.