- Mango + DankMaterialShell desktop option (desktop.profile = "mango") through the GURU overlay with scoped ~amd64 keywords, a sealed session entry point and a dwl fallback session (ADR 0009). - Coherent rollback/recovery: coordinated @ + @var snapshots, fstab and BLS root-subvolume rewriting, matching-kernel selection, snapshot recovery entry, retention/space gates and sys undo (ADR 0007). - Installed assistant (sys ask) with an OS-level systemd sandbox boundary and fail-closed typed plans; resolved package effects bound to approval. - Nix build isolation enforced (sandbox, no fallback) plus lifecycle/cleanup. - Desktop helpers (menu/lock/idle), transaction log, release gates with atomic publish and signing helpers, and a Debian-Live appliance VM harness.
4.6 KiB
ADR 0007: Coherent rollback and recovery
- Status: Accepted
- Date: 2026-09-21
Context
Btrfs snapshots and transaction history existed, but rollback was deliberately unimplemented. The proven layout separates the filesystems that must move together:
GPT: 1 GiB FAT32 ESP (mounted /efi)
LUKS2 -> Btrfs: @ / @home /home @var /var @snapshots /.snapshots @swap /swap
Three facts make a naive rollback wrong:
/varis a separate subvolume. The Portage package database (/var/db/pkg), Portage state (/var/lib/portage), the update state and the snapshot/transaction history all live in@var. Restoring@alone leaves the package manager describing packages that do not exist in the restored root.- The kernel command line carries an explicit root subvolume.
rootflags=subvol=@overrides the Btrfs default subvolume, so changing the default alone does not change what boots. - The kernel and initramfs live on the ESP, which is not snapshotted.
A restored older root has older
/lib/modules; it must boot a kernel whose modules match, or it will fail to load drivers.
The ESP also holds the BLS entries, so a recovery entry that reuses the live root's kernel/options still depends on the root being healthy.
Decision
Rollback is a coordinated operation over four things, implemented in
phoebus-executor::rollback and planned by phoebus-core::rollback:
-
Snapshot set. A recovery point captures
@and the separately mounted@varunder one timestamp/operation name./homeis intentionally excluded so rollback never discards user documents./nixis inside@, so the Nix store rolls back with the OS package state (consistent by construction). -
Root generation. Rollback creates a writable clone of the chosen snapshot (
@restore-<utc>) and, when present,@varrestore-<utc>. The running root is preserved by the pre-rollback snapshot; the previous root is also offered as a boot entry. -
Boot arguments, in both places. The restored root's
/etc/fstabis rewritten to mountsubvol=<new root>andsubvol=<new var>, and the BLS entries are rewritten (rootflags=subvol=<new root>). The Btrfs default subvolume is set to match, as a fallback for a mount without an argument. -
Kernel/initramfs pairing. The restored root's
/lib/modulesversions are used to select the newest BLS entry with matching modules. A "Phoebus OS (previous)" entry and a "Phoebus Recovery" entry that boots the chosen snapshot directly intorescue.targetare written, so recovery does not depend on the live root.
State machine and interruption safety
var/lib/phoebus/rollback-state.toml records Idle | Planned | Activated | Verified | Failed. The safety snapshot is taken first; boot entries are
switched last; a sync runs before completion. An interruption therefore leaves
the system booting a known root, and re-running resumes (an already-staged
restored root is detected and reused). sys undo status promotes Activated
to Verified once the restored subvolume is the running root.
Retention and space
Retention keeps the newest DEFAULT_KEEP_SNAPSHOTS (8) committed snapshots and
never prunes the newest, an in-flight rollback target, or Created/Failed
records. Rollback refuses to start unless at least 1 GiB is free on the target.
Exposure
sys undo is the only acceptance test for a rollback. sys undo restores the
newest committed snapshot; sys undo <id> selects one; sys undo status
reports state. It requires root and a reviewed typed action, like every other
high-risk mutation.
Consequences
- More subvolumes accumulate (
@restore-*,@varrestore-*); retention bounds them and the operator can delete them explicitly. - A rollback is a reboot-level operation; it changes the next boot, not the running system. This is deliberate: swapping the mounted root in place is not safe.
- The path is unit-tested for the command sequence, fstab/boot-entry rewriting,
kernel selection, interruption resume and retention. End-to-end VM
verification (broken update, rollback, detached boot, package-state check) is
the acceptance gate recorded in
docs/STATUS.md.
Alternatives rejected
- Changing only the Btrfs default subvolume. Rejected: the explicit
rootflags=subvol=wins, so this does not boot the restored root. - Renaming the mounted root subvolume. Rejected: unsafe while it is the live root.
- Snapshotting the ESP. Impossible (vfat, not Btrfs); handled instead by selecting a matching kernel entry and keeping a known-good entry.