phoebus_os/docs/VM-BOOT-VERIFICATION.md
Phoebus f8b34635c7 docs: record the mandatory-AI gate, local provider and portals
Note the rebuilt appliance ISO (fec8e312) and the new desktop portals; add
Nix packaging as the top next task since it was deliberately not faked.
2026-09-20 14:43:34 -07:00

16 KiB
Raw Blame History

Disposable UEFI VM boot verification

This records the first successful end-to-end install and boot of Phoebus OS against a disposable qcow2 disk. No physical disk was ever touched.

Exact tested path

Host (Ubuntu, no root required):

  • QEMU qemu-system-x86_64 (extracted locally under /tmp/opencode/qemudeb), OVMF (OVMF_CODE_4M.fd + OVMF_VARS_4M.fd).
  • Target disk: /tmp/opencode/vm/work/target.qcow2, 30 GiB, virtio-blk (/dev/vda in the guest).
  • Live boot: the reference ISO's own boot/gentoo + boot/gentoo.igz, with -append "root=live:CDLABEL=gentoo-amd64-livegui rd.live.dir=/ rd.live.squashimg=image.squashfs cdroot console=ttyS0,115200".
  • A host python3 -m http.server serves the installer tarball, the pinned stage3 and the test config to the guest over QEMU user networking.

Phase 1 (install):

  1. Boot the live ISO headless; get a root shell on the serial console.
  2. Download the installer and test config from the host.
  3. Write secrets to 0600 files and run the installer binary (the original run used the Bash prototype; the harness now serves phoebus-installer): phoebus-installer --config test.conf --install --yes --luks-passphrase-file FILE --user-password-file FILE
  4. Stages executed in order: preflight, storage, stage3, fstab, portage, base packages, kernel, bootloader, users, services, desktop, finalize.

Phase 2 (boot the disk alone, ISO detached):

  1. Power off, then start a new QEMU with no ISO and no -kernel, boot order=c, same OVMF NVRAM.
  2. Firmware runs \EFI\systemd\systemd-bootx64.efi; systemd-boot boots the "Gentoo Linux" BLS entry from the ESP.
  3. systemd-cryptsetup prompts for the LUKS2 passphrase, unlocks /dev/vda2, mounts the Btrfs @ subvolume as /.
  4. systemd (PID 1) starts, serial getty appears, user login succeeds.

Verified results

  • Firmware booted the disk's own systemd-boot (Boot000C "Linux Boot Manager").
  • LUKS2 passphrase prompt and successful unlock.
  • Btrfs subvolumes mounted: /, /home, /var, /.snapshots, /swap (later fix), ESP at /efi.
  • cat /proc/1/comm -> systemd.
  • systemd-analyze: 1.442s kernel + 5.755s initrd + 2.942s userspace.
  • Login as phoebus with the configured password; groups: wheel audio video render input plugdev power network libinput.
  • Networking up (enp0s2 10.0.2.15/24).
  • systemctl --failed: exactly one unit, swap-swapfile.swap (fixed after this run; see below).

Bugs found and fixed during this work

  1. Subvolumes created after mounting @ - storage_prepare mounted subvol=@ before @ existed (mount exit 32). Added btrfs_create_layout (mount top-level, create subvols, unmount, then mount layout) and a regression test (tests/test_btrfs_order.sh).
  2. Generated fstab overwritten by stage3 - fstab/crypttab were generated during storage, then the stage3 tarball replaced /etc/fstab. Moved generation to a new fstab_generate stage after extraction.
  3. /var/tmp absent in the target - systemd's kernel-install needs a temporary directory; /var/tmp was missing. The fstab stage now creates /var/tmp, /tmp, /root, /home, and chroot commands set TMPDIR=/tmp.
  4. installkernel needs uki_generator even for BLS - with an explicit /etc/kernel/install.conf, 05-check-config.install always requires a uki_generator. BLS now writes uki_generator=none.
  5. dracut could not build the crypt module - the target lacked sys-fs/cryptsetup. Added sys-fs/cryptsetup and sys-fs/btrfs-progs to the base package set.
  6. No systemd-cryptsetup - sys-apps/systemd was built without the cryptsetup USE flag, so the encrypted root could not be unlocked. Added cryptsetup (and policykit) to the systemd package.use entry.
  7. Kernel command line used unnamed LUKS - rd.luks.uuid yields luks-<uuid>, not the configured name. Switched to rd.luks.name=<uuid>=<name>.
  8. Dracut UKI did not embed the command line - the dracut uki_generator produced a UKI with no .cmdline, so it booted with no root=. Deferred UKI in favour of a reliable BLS entry carrying options. UKI remains future work.
  9. fstab did not mount @swap - the swapfile unit failed because /swap was never the @swap subvolume. Added the subvol=@swap fstab entry.
  10. local root="$1" dir="$root/..." class bug - reintroduced in boot_write_dracut_conf; split into two local statements. This class of bug is now covered by the regression suite.
  11. Wrong package atoms - gui-apps/fuzzel (not in Gentoo) -> gui-apps/wofi, app-misc/brightnessctl (not packaged) removed, app-crypt/gnome-keyring -> gnome-base/gnome-keyring, gui-apps/polkit-gnome -> gnome-extra/polkit-gnome.
  12. Package policy compiled too much - unknown packages defaulted to source. balanced now compiles cheap builds and uses signed Gentoo binaries for everything else, which cut installer time substantially.

Rust installer parity (same VM path)

The Rust installer (installer-rs) reproduced the entire path on a fresh disposable qcow2 disk using the static binary phoebus-installer (x86_64-unknown-linux-gnu, crt-static):

  • Same config (legacy flat format), same pinned stage3 URL + SHA-256.
  • INSTALL_RC=0; stages: storage, stage3, fstab, portage, base packages, kernel, bootloader, users, services, desktop (none), finalize.
  • Policy decisions matched Bash: sudo/networkmanager/polkit binary, cryptsetup/btrfs-progs source.
  • The installer recorded 81 then 97 measured compile times from Portage's emerge.log (real data, not estimates).
  • NO_SECRET_LEAK from the log secret scan.

Booted with the ISO detached:

  • UEFI firmware ran systemd-boot; BLS entry booted.
  • LUKS2 unlock, Btrfs @ root, /home, /var, /swap, ESP at /efi.
  • cat /proc/1/comm -> systemd.
  • systemctl is-system-running -> running (not degraded).
  • systemctl --failed -> 0 loaded units listed.
  • systemd-analyze: 1.378s kernel + 5.238s initrd + 4.114s userspace = 10.730s.
  • Login as phoebus succeeded; networking up (10.0.2.15/24).

This is the point at which the Rust installer satisfies the parity criteria and may become the primary installer. The Bash implementation remains the reference/fallback until the desktop milestone is also verified.

Desktop and Flatpak (Rust installer, verified)

The Rust installer, running the desktop profile, installed the full Wayland desktop and Flatpak successfully (INSTALL_RC=0; 326 measured compile times recorded). Key change: a desktop install now selects the supported stable Gentoo profile default/linux/amd64/23.0/desktop/systemd instead of reconstructing X/Wayland/dbus USE by hand from the minimal systemd profile. This resolved the earlier cairo[X]/libglvnd[X]/libxkbcommon[X] conflicts, and sys-fs/{btrfs-progs,cryptsetup} -man avoided a Sphinx/Pillow circular dependency.

Booted with the ISO detached and logged in:

  • systemctl is-system-running -> running; systemctl --failed -> 0 units.
  • flatpak --version -> Flatpak 1.18.1; flatpak remotes -> flathub configured.
  • Binaries present and runnable: dwl 0.7, Waybar v0.15.0, wofi 1.4.1, mako, swaybg, pipewire, wireplumber, grim, wl-clipboard.

Session verification. The VM's serial session has no logind seat, so wlroots' DRM backend correctly refuses (Failed to start a DRM session). This is a headless-harness limitation, not an installer defect. Using WLR_BACKENDS=headless WLR_RENDERER=pixman:

  • dwl started (pid observed), created the wayland-0 socket.
  • mako, waybar, and swaybg started and connected (swaybg found output HEADLESS-1).
  • wl-copy/wl-paste round-tripped phoebus-clip.
  • grim /tmp/shot.png produced a 14401-byte PNG, proving the compositor renders and captures.
  • wpctl status reported the PipeWire graph; systemctl --user --failed -> 0.
  • waybar logged a non-fatal warning about a missing power-menu file and power-profiles-daemon.

The installer writes /usr/local/bin/phoebus-session, which starts waybar, mako and swaybg and then execs dwl, with a software-render fallback when no render node is present.

Official harness on the static binary (Bash migration, Phase 2)

The committed harness (installer/tests/vm/serial-driver.py + installer/tests/vm/install.sh) now drives the static Rust binary and no longer runs the Bash installer. Re-verified end to end on a fresh disposable qcow2 (24 GiB, virtio-blk /dev/vda):

# Build:
cd installer-rs && RUSTFLAGS="-C target-feature=+crt-static" \
  cargo build --release -p phoebus-installer --target x86_64-unknown-linux-gnu

# Install in the disposable VM, then boot the disk detached:
python3 installer/tests/vm/serial-driver.py install
python3 installer/tests/vm/serial-driver.py boot

Install (INSTALL-RC=0):

  • The guest fetched /tmp/phoebus-installer and printed phoebus-installer 0.1.0 (the Rust binary, not Bash).
  • Stages completed in proven order: storage_prepare, stage_install, fstab_generate, portage_configure, packages_base, kernel_install, boot_install, users_configure, services_configure, desktop_install (none), finalize.
  • Policy matched Bash: crypto/btrfs-progs source build, sudo/ networkmanager/polkit binary.
  • NO_SECRET_LEAK; the run powered off cleanly.

Boot with the ISO detached (serial-driver.py boot):

  • UEFI firmware ran \EFI\systemd\systemd-bootx64.efi (Boot0002 "Linux Boot Manager"); the BLS "Gentoo Linux" entry booted.
  • systemd-cryptsetup prompted for the LUKS2 passphrase and unlocked phoebus-root (cryptroot).
  • Login as phoebus succeeded.
  • cat /proc/1/comm -> systemd.
  • systemctl is-system-running -> running.
  • systemctl --failed -> 0 loaded units listed.
  • Networking: enp0s2 UP 10.0.2.15/24.
  • Mounts: / (btrfs @), /home, /var, /.snapshots, /swap, ESP /efi; vda2 is crypto_LUKS.

This satisfies the Phase 2 exit gate of docs/BASH-MIGRATION.md: the only proven boot path no longer depends on the retired Bash installer.

Snapshots and safe updates (four-workstream session)

Verified on the installed Btrfs system from the static-binary harness, driving the new sys binary fetched into the guest:

  • sys history before -> no transaction history.
  • sys -y --action '{"action":"create_snapshot","operation":"vm test"}' -> protective snapshot: @-20260920T114917-vm-test.
  • sys history after -> @-20260920T114917-vm-test Committed vm test.
  • btrfs subvolume list / shows @snapshots/@-20260920T114917-vm-test as a real subvolume (read-only snapshot).
  • sys -y prepare-updates fetched distfiles successfully; --buildpkgonly was refused by Portage with --buildpkgonly requires all dependencies to be merged, which the installer now records as status: Ready with the reason (distfiles fetched; binary prebuild skipped: ...) instead of a hard failure.
  • sys inspect-updates reports the state and timestamp.

Unit regression tests cover the failure paths that a VM cannot easily force: corrupt snapshot history is an error, and a high-risk action aborts when the protective snapshot fails.

Desktop session on an installed VM

A full dwl desktop install was built with the static Rust binary (DESKTOP_PROFILE=dwl, FLATPAK=no) and booted with the ISO detached: INSTALL_RC=0, all stages, NO_SECRET_LEAK, systemctl is-system-running -> running, 0 failed units.

Running the installed /usr/local/bin/phoebus-session under the headless wlroots backend (the serial VM has no logind seat):

  • the wayland-0 socket is created before any helper starts;
  • after startup the process tree is dwl, waybar, mako, swaybg -c #1e1e2e and /usr/libexec/polkit-gnome-authentication-agent-1 (the polkit agent, which the previous script never started);
  • dwl reports the HEADLESS-1 output;
  • after dwl exits, all session processes are reaped.

Installed and on PATH: dwl, waybar, mako, swaybg, kitty, wofi, Xwayland. The polkit agent lives at /usr/libexec/polkit-gnome-authentication-agent-1; the session script's fallback finds it because it is not on PATH.

This is headless verification. A physical VT / real logind seat remains to be exercised interactively (docs/DEVELOPMENT.md -> "Physical VT / real-seat check").

Signed binary reuse analysis

The desktop transaction's installer log (emerge grouping) shows the policy already requests signed binaries for the expensive set:

  • source group: sys-fs/cryptsetup sys-fs/btrfs-progs;
  • binary groups: base tools, gentoo-kernel-bin/firmware, and the dwl/waybar/ wofi/mako/sway*/gtk-layer-shell/xwayland set.

The desktop install in this session took ~1h49m wall time (12:01–13:50 UTC) on 8 vCPUs. Most of that is Portage falling back to source because the OpenRC/ hardened binhost cannot satisfy the systemd/desktop USE set, even though the policy requested binaries. Two comparable data points are now recorded: the base-only install took ~44 min and the desktop install ~1h49m. The code changes here remove forced flags but cannot raise reuse without a matching binhost, so no new time saving is claimed.

The remaining source builds are caused by the official Gentoo binhost being built for the OpenRC/hardened profile (PROFILE: default/linux/amd64/17.0/ hardened), which cannot satisfy a systemd/desktop USE set. The safe changes made here are: stop forcing the unused installkernel uki flag, remove the duplicate PipeWire and empty WirePlumber package.use lines, and derive the binhost URI from the selected profile when it is not set. Signature verification remains enabled. Publishing a matching systemd/desktop binhost is the remaining step; no time estimate is claimed beyond the recorded ~26 min.

Debian-Live installer appliance (actual ISO)

An appliance ISO was built without root by installer/live/build.sh (Debian-Live 13.7 standard base + /phoebus overlay). The full-install evidence below was recorded with build c0105bc9a282fddcbefab42baa577ea99843060f00fa45a75bcbc8358186bfd5.

The current build is fec8e312bfe641c423f2656970ae4271a1544b3a6107b407a8a29cb71f8850be; it carries the mandatory-AI install gate (--no-ai/--legacy break-glass), the local model provider, and the desktop portals. It was not re-run through the full install in this session (resource constraints); treat the earlier run as the boot/install evidence and this as the current artifact.

Firmware boot (UEFI/OVMF, no -kernel; the ISO's own grub chain):

  • auto-boots the live system on the serial console;
  • login user / live (passwordless sudo);
  • the overlaid tools run: cryptsetup 2.7.5, sgdisk 1.0.10, partprobe 3.6, mkfs.btrfs 6.14, mkfs.fat.

Full install to a fresh 24 GiB qcow2 driven from the appliance:

  • INSTALL_RC=0, all stages, NO_SECRET_LEAK.

Detached boot of the installed disk (ISO removed):

  • systemd-boot -> LUKS2 unlock -> phoebus login;
  • systemctl is-system-running -> running; 0 loaded units listed;
  • Gentoo-only: Gentoo Base System release 2.18, Portage 3.0.82.2, no-apt; systemd PID 1;
  • cryptroot Btrfs mounted at /, /home, /var, /.snapshots, /swap, ESP /efi.

This is Bash-free and Gentoo-target-only: the appliance contributes tooling, never configuration.

Open items found in this run

  • The target came up with the default gentoo hostname (empty /etc/hostname) even though the plan/config specified phoebus, unlike the earlier Gentoo-appliance installs. Needs a focused look (possible stage3 or ordering difference).
  • The first detached boot reported enp0s2 DOWN; DHCP timing was not re-checked. The earlier Gentoo-appliance boot had enp0s2 UP 10.0.2.15/24.

Neither affects the proven Gentoo-appliance boot path; both are recorded rather than hidden.

Known remaining work

  • The graphical session was verified with the headless wlroots backend (no logind seat in the serial harness). A physical VT / seat path is architected (phoebus-session) but was not exercised on real hardware.
  • NVIDIA proprietary support is architecturally allowed but NOT VERIFIED.
  • UKI is deferred (BLS is used).
  • The installer rebuilds sys-apps/systemd from source because boot/policykit/cryptsetup USE differ from the binhost. Shipping a matching systemd binpkg is a future optimization.
  • Desktop USE still does not match the signed binhost, so the desktop transaction compiles a large amount from source (~26 minutes for the desktop package set on 8 vCPUs in the test VM).