347 lines
16 KiB
Markdown
347 lines
16 KiB
Markdown
# Disposable UEFI VM boot verification
|
||
|
||
This records the first successful end-to-end install and boot of Phoebus OS
|
||
against a disposable qcow2 disk. No physical disk was ever touched.
|
||
|
||
## Exact tested path
|
||
|
||
Host (Ubuntu, no root required):
|
||
|
||
- QEMU `qemu-system-x86_64` (extracted locally under `/tmp/opencode/qemudeb`),
|
||
OVMF (`OVMF_CODE_4M.fd` + `OVMF_VARS_4M.fd`).
|
||
- Target disk: `/tmp/opencode/vm/work/target.qcow2`, 30 GiB, virtio-blk
|
||
(`/dev/vda` in the guest).
|
||
- Live boot: the reference ISO's own `boot/gentoo` + `boot/gentoo.igz`, with
|
||
`-append "root=live:CDLABEL=gentoo-amd64-livegui rd.live.dir=/
|
||
rd.live.squashimg=image.squashfs cdroot console=ttyS0,115200"`.
|
||
- A host `python3 -m http.server` serves the installer tarball, the pinned
|
||
stage3 and the test config to the guest over QEMU user networking.
|
||
|
||
Phase 1 (install):
|
||
|
||
1. Boot the live ISO headless; get a root shell on the serial console.
|
||
2. Download the installer and test config from the host.
|
||
3. Write secrets to `0600` files and run the installer binary (the original
|
||
run used the Bash prototype; the harness now serves `phoebus-installer`):
|
||
`phoebus-installer --config test.conf --install --yes
|
||
--luks-passphrase-file FILE --user-password-file FILE`
|
||
4. Stages executed in order: `preflight, storage, stage3, fstab,
|
||
portage, base packages, kernel, bootloader, users, services, desktop,
|
||
finalize`.
|
||
|
||
Phase 2 (boot the disk alone, ISO detached):
|
||
|
||
5. Power off, then start a new QEMU with **no ISO and no `-kernel`**, boot
|
||
`order=c`, same OVMF NVRAM.
|
||
6. Firmware runs `\EFI\systemd\systemd-bootx64.efi`; systemd-boot boots the
|
||
"Gentoo Linux" BLS entry from the ESP.
|
||
7. `systemd-cryptsetup` prompts for the LUKS2 passphrase, unlocks
|
||
`/dev/vda2`, mounts the Btrfs `@` subvolume as `/`.
|
||
8. systemd (PID 1) starts, serial getty appears, user login succeeds.
|
||
|
||
## Verified results
|
||
|
||
- Firmware booted the disk's own systemd-boot (Boot000C "Linux Boot Manager").
|
||
- LUKS2 passphrase prompt and successful unlock.
|
||
- Btrfs subvolumes mounted: `/`, `/home`, `/var`, `/.snapshots`, `/swap`
|
||
(later fix), ESP at `/efi`.
|
||
- `cat /proc/1/comm` -> `systemd`.
|
||
- `systemd-analyze`: 1.442s kernel + 5.755s initrd + 2.942s userspace.
|
||
- Login as `phoebus` with the configured password; groups:
|
||
`wheel audio video render input plugdev power network libinput`.
|
||
- Networking up (`enp0s2 10.0.2.15/24`).
|
||
- `systemctl --failed`: exactly one unit, `swap-swapfile.swap` (fixed after
|
||
this run; see below).
|
||
|
||
## Bugs found and fixed during this work
|
||
|
||
1. **Subvolumes created after mounting `@`** - `storage_prepare` mounted
|
||
`subvol=@` before `@` existed (mount exit 32). Added `btrfs_create_layout`
|
||
(mount top-level, create subvols, unmount, then mount layout) and a
|
||
regression test (`tests/test_btrfs_order.sh`).
|
||
2. **Generated fstab overwritten by stage3** - fstab/crypttab were generated
|
||
during storage, then the stage3 tarball replaced `/etc/fstab`. Moved
|
||
generation to a new `fstab_generate` stage after extraction.
|
||
3. **`/var/tmp` absent in the target** - systemd's `kernel-install` needs a
|
||
temporary directory; `/var/tmp` was missing. The fstab stage now creates
|
||
`/var/tmp`, `/tmp`, `/root`, `/home`, and chroot commands set
|
||
`TMPDIR=/tmp`.
|
||
4. **installkernel needs `uki_generator` even for BLS** - with an explicit
|
||
`/etc/kernel/install.conf`, `05-check-config.install` always requires a
|
||
`uki_generator`. BLS now writes `uki_generator=none`.
|
||
5. **dracut could not build the `crypt` module** - the target lacked
|
||
`sys-fs/cryptsetup`. Added `sys-fs/cryptsetup` and `sys-fs/btrfs-progs` to
|
||
the base package set.
|
||
6. **No `systemd-cryptsetup`** - `sys-apps/systemd` was built without the
|
||
`cryptsetup` USE flag, so the encrypted root could not be unlocked. Added
|
||
`cryptsetup` (and `policykit`) to the systemd package.use entry.
|
||
7. **Kernel command line used unnamed LUKS** - `rd.luks.uuid` yields
|
||
`luks-<uuid>`, not the configured name. Switched to
|
||
`rd.luks.name=<uuid>=<name>`.
|
||
8. **Dracut UKI did not embed the command line** - the dracut `uki_generator`
|
||
produced a UKI with no `.cmdline`, so it booted with no `root=`. Deferred
|
||
UKI in favour of a reliable BLS entry carrying `options`. UKI remains
|
||
future work.
|
||
9. **fstab did not mount `@swap`** - the swapfile unit failed because `/swap`
|
||
was never the `@swap` subvolume. Added the `subvol=@swap` fstab entry.
|
||
10. **`local root="$1" dir="$root/..."` class bug** - reintroduced in
|
||
`boot_write_dracut_conf`; split into two `local` statements. This class of
|
||
bug is now covered by the regression suite.
|
||
11. **Wrong package atoms** - `gui-apps/fuzzel` (not in Gentoo) -> `gui-apps/wofi`,
|
||
`app-misc/brightnessctl` (not packaged) removed,
|
||
`app-crypt/gnome-keyring` -> `gnome-base/gnome-keyring`,
|
||
`gui-apps/polkit-gnome` -> `gnome-extra/polkit-gnome`.
|
||
12. **Package policy compiled too much** - unknown packages defaulted to
|
||
source. `balanced` now compiles cheap builds and uses signed Gentoo
|
||
binaries for everything else, which cut installer time substantially.
|
||
|
||
## Rust installer parity (same VM path)
|
||
|
||
The Rust installer (`installer-rs`) reproduced the entire path on a fresh
|
||
disposable qcow2 disk using the static binary
|
||
`phoebus-installer` (`x86_64-unknown-linux-gnu`, crt-static):
|
||
|
||
- Same config (legacy flat format), same pinned stage3 URL + SHA-256.
|
||
- `INSTALL_RC=0`; stages: storage, stage3, fstab, portage, base packages,
|
||
kernel, bootloader, users, services, desktop (none), finalize.
|
||
- Policy decisions matched Bash: `sudo`/`networkmanager`/`polkit` binary,
|
||
`cryptsetup`/`btrfs-progs` source.
|
||
- The installer recorded **81 then 97 measured compile times** from Portage's
|
||
`emerge.log` (real data, not estimates).
|
||
- `NO_SECRET_LEAK` from the log secret scan.
|
||
|
||
Booted with the ISO detached:
|
||
|
||
- UEFI firmware ran systemd-boot; BLS entry booted.
|
||
- LUKS2 unlock, Btrfs `@` root, `/home`, `/var`, `/swap`, ESP at `/efi`.
|
||
- `cat /proc/1/comm` -> `systemd`.
|
||
- `systemctl is-system-running` -> **running** (not degraded).
|
||
- `systemctl --failed` -> **0 loaded units listed**.
|
||
- `systemd-analyze`: 1.378s kernel + 5.238s initrd + 4.114s userspace =
|
||
10.730s.
|
||
- Login as `phoebus` succeeded; networking up (`10.0.2.15/24`).
|
||
|
||
This is the point at which the Rust installer satisfies the parity criteria and
|
||
may become the primary installer. The Bash implementation remains the
|
||
reference/fallback until the desktop milestone is also verified.
|
||
|
||
## Desktop and Flatpak (Rust installer, verified)
|
||
|
||
The Rust installer, running the desktop profile, installed the full Wayland
|
||
desktop and Flatpak successfully (`INSTALL_RC=0`; 326 measured compile times
|
||
recorded). Key change: a desktop install now selects the supported stable
|
||
Gentoo profile `default/linux/amd64/23.0/desktop/systemd` instead of
|
||
reconstructing X/Wayland/dbus USE by hand from the minimal systemd profile.
|
||
This resolved the earlier `cairo[X]`/`libglvnd[X]`/`libxkbcommon[X]`
|
||
conflicts, and `sys-fs/{btrfs-progs,cryptsetup} -man` avoided a
|
||
Sphinx/Pillow circular dependency.
|
||
|
||
Booted with the ISO detached and logged in:
|
||
|
||
- `systemctl is-system-running` -> `running`; `systemctl --failed` -> 0 units.
|
||
- `flatpak --version` -> **Flatpak 1.18.1**; `flatpak remotes` -> **flathub
|
||
configured**.
|
||
- Binaries present and runnable: `dwl 0.7`, `Waybar v0.15.0`, `wofi 1.4.1`,
|
||
`mako`, `swaybg`, `pipewire`, `wireplumber`, `grim`, `wl-clipboard`.
|
||
|
||
Session verification. The VM's serial session has **no logind seat**, so
|
||
wlroots' DRM backend correctly refuses (`Failed to start a DRM session`). This
|
||
is a headless-harness limitation, not an installer defect. Using
|
||
`WLR_BACKENDS=headless WLR_RENDERER=pixman`:
|
||
|
||
- `dwl` started (pid observed), created the `wayland-0` socket.
|
||
- `mako`, `waybar`, and `swaybg` started and connected (`swaybg` found output
|
||
`HEADLESS-1`).
|
||
- `wl-copy`/`wl-paste` round-tripped `phoebus-clip`.
|
||
- `grim /tmp/shot.png` produced a **14401-byte PNG**, proving the compositor
|
||
renders and captures.
|
||
- `wpctl status` reported the PipeWire graph; `systemctl --user --failed` -> 0.
|
||
- `waybar` logged a non-fatal warning about a missing power-menu file and
|
||
power-profiles-daemon.
|
||
|
||
The installer writes `/usr/local/bin/phoebus-session`, which starts waybar,
|
||
mako and swaybg and then execs dwl, with a software-render fallback when no
|
||
render node is present.
|
||
|
||
## Official harness on the static binary (Bash migration, Phase 2)
|
||
|
||
The committed harness (`installer/tests/vm/serial-driver.py` +
|
||
`installer/tests/vm/install.sh`) now drives the **static Rust binary** and no
|
||
longer runs the Bash installer. Re-verified end to end on a fresh disposable
|
||
qcow2 (24 GiB, virtio-blk `/dev/vda`):
|
||
|
||
```sh
|
||
# Build:
|
||
cd installer-rs && RUSTFLAGS="-C target-feature=+crt-static" \
|
||
cargo build --release -p phoebus-installer --target x86_64-unknown-linux-gnu
|
||
|
||
# Install in the disposable VM, then boot the disk detached:
|
||
python3 installer/tests/vm/serial-driver.py install
|
||
python3 installer/tests/vm/serial-driver.py boot
|
||
```
|
||
|
||
Install (`INSTALL-RC=0`):
|
||
|
||
- The guest fetched `/tmp/phoebus-installer` and printed
|
||
`phoebus-installer 0.1.0` (the Rust binary, not Bash).
|
||
- Stages completed in proven order: `storage_prepare`, `stage_install`,
|
||
`fstab_generate`, `portage_configure`, `packages_base`, `kernel_install`,
|
||
`boot_install`, `users_configure`, `services_configure`, `desktop_install`
|
||
(none), `finalize`.
|
||
- Policy matched Bash: `crypto`/`btrfs-progs` source build, `sudo`/
|
||
`networkmanager`/`polkit` binary.
|
||
- `NO_SECRET_LEAK`; the run powered off cleanly.
|
||
|
||
Boot with the ISO detached (`serial-driver.py boot`):
|
||
|
||
- UEFI firmware ran `\EFI\systemd\systemd-bootx64.efi` (Boot0002 "Linux Boot
|
||
Manager"); the BLS "Gentoo Linux" entry booted.
|
||
- `systemd-cryptsetup` prompted for the LUKS2 passphrase and unlocked
|
||
`phoebus-root (cryptroot)`.
|
||
- Login as `phoebus` succeeded.
|
||
- `cat /proc/1/comm` -> `systemd`.
|
||
- `systemctl is-system-running` -> **running**.
|
||
- `systemctl --failed` -> **0 loaded units listed**.
|
||
- Networking: `enp0s2 UP 10.0.2.15/24`.
|
||
- Mounts: `/` (btrfs `@`), `/home`, `/var`, `/.snapshots`, `/swap`, ESP
|
||
`/efi`; `vda2` is `crypto_LUKS`.
|
||
|
||
This satisfies the Phase 2 exit gate of `docs/BASH-MIGRATION.md`: the only
|
||
proven boot path no longer depends on the retired Bash installer.
|
||
|
||
## Snapshots and safe updates (four-workstream session)
|
||
|
||
Verified on the installed Btrfs system from the static-binary harness, driving
|
||
the new `sys` binary fetched into the guest:
|
||
|
||
- `sys history` before -> `no transaction history`.
|
||
- `sys -y --action '{"action":"create_snapshot","operation":"vm test"}'` ->
|
||
`protective snapshot: @-20260920T114917-vm-test`.
|
||
- `sys history` after -> `@-20260920T114917-vm-test Committed vm test`.
|
||
- `btrfs subvolume list /` shows `@snapshots/@-20260920T114917-vm-test` as a
|
||
real subvolume (read-only snapshot).
|
||
- `sys -y prepare-updates` fetched distfiles successfully; `--buildpkgonly` was
|
||
refused by Portage with `--buildpkgonly requires all dependencies to be
|
||
merged`, which the installer now records as `status: Ready` with the reason
|
||
(`distfiles fetched; binary prebuild skipped: ...`) instead of a hard failure.
|
||
- `sys inspect-updates` reports the state and timestamp.
|
||
|
||
Unit regression tests cover the failure paths that a VM cannot easily force:
|
||
corrupt snapshot history is an error, and a high-risk action aborts when the
|
||
protective snapshot fails.
|
||
|
||
## Desktop session on an installed VM
|
||
|
||
A full dwl desktop install was built with the static Rust binary
|
||
(`DESKTOP_PROFILE=dwl`, `FLATPAK=no`) and booted with the ISO detached:
|
||
`INSTALL_RC=0`, all stages, `NO_SECRET_LEAK`, `systemctl is-system-running` ->
|
||
`running`, `0 failed units`.
|
||
|
||
Running the installed `/usr/local/bin/phoebus-session` under the headless
|
||
wlroots backend (the serial VM has no logind seat):
|
||
|
||
- the `wayland-0` socket is created before any helper starts;
|
||
- after startup the process tree is `dwl`, `waybar`, `mako`,
|
||
`swaybg -c #1e1e2e` and `/usr/libexec/polkit-gnome-authentication-agent-1`
|
||
(the polkit agent, which the previous script never started);
|
||
- dwl reports the `HEADLESS-1` output;
|
||
- after `dwl` exits, **all session processes are reaped**.
|
||
|
||
Installed and on `PATH`: `dwl`, `waybar`, `mako`, `swaybg`, `kitty`, `wofi`,
|
||
`Xwayland`. The polkit agent lives at
|
||
`/usr/libexec/polkit-gnome-authentication-agent-1`; the session script's
|
||
fallback finds it because it is not on `PATH`.
|
||
|
||
This is headless verification. A physical VT / real logind seat remains to be
|
||
exercised interactively (`docs/DEVELOPMENT.md` -> "Physical VT / real-seat
|
||
check").
|
||
|
||
## Signed binary reuse analysis
|
||
|
||
The desktop transaction's installer log (`emerge` grouping) shows the policy
|
||
already requests signed binaries for the expensive set:
|
||
|
||
- source group: `sys-fs/cryptsetup sys-fs/btrfs-progs`;
|
||
- binary groups: base tools, `gentoo-kernel-bin`/firmware, and the dwl/waybar/
|
||
wofi/mako/sway*/gtk-layer-shell/xwayland set.
|
||
|
||
The desktop install in this session took ~1h49m wall time (12:01–13:50 UTC) on
|
||
8 vCPUs. Most of that is Portage falling back to source because the OpenRC/
|
||
hardened binhost cannot satisfy the systemd/desktop USE set, even though the
|
||
policy requested binaries. Two comparable data points are now recorded: the
|
||
base-only install took ~44 min and the desktop install ~1h49m. The code changes
|
||
here remove forced flags but cannot raise reuse without a matching binhost, so
|
||
no new time saving is claimed.
|
||
|
||
The remaining source builds are caused by the **official Gentoo binhost being
|
||
built for the OpenRC/hardened profile** (`PROFILE: default/linux/amd64/17.0/
|
||
hardened`), which cannot satisfy a systemd/desktop USE set. The safe changes
|
||
made here are: stop forcing the unused `installkernel uki` flag, remove the
|
||
duplicate PipeWire and empty WirePlumber `package.use` lines, and derive the
|
||
binhost URI from the selected profile when it is not set. Signature
|
||
verification remains enabled. Publishing a matching systemd/desktop binhost is
|
||
the remaining step; no time estimate is claimed beyond the recorded ~26 min.
|
||
|
||
## Debian-Live installer appliance (actual ISO)
|
||
|
||
An appliance ISO was built **without root** by `installer/live/build.sh`
|
||
(Debian-Live 13.7 standard base + `/phoebus` overlay). The full-install
|
||
evidence below was recorded with build
|
||
`c0105bc9a282fddcbefab42baa577ea99843060f00fa45a75bcbc8358186bfd5`.
|
||
|
||
The current build is
|
||
`690936d15912bdef8eb0edb5932e3a845084b492073148687090a79ddb991078`; it carries
|
||
the mandatory-AI install gate (`--no-ai`/`--legacy` break-glass), the `local`
|
||
model provider, the desktop portals, and the gated (`packages.nix=no`) Nix
|
||
bootstrap. It was **not** re-run through the full install in this session
|
||
(resource constraints); treat the earlier run as the boot/install evidence and
|
||
this as the current artifact.
|
||
|
||
Firmware boot (UEFI/OVMF, no `-kernel`; the ISO's own grub chain):
|
||
|
||
- auto-boots the live system on the serial console;
|
||
- login `user` / `live` (passwordless sudo);
|
||
- the overlaid tools run: `cryptsetup 2.7.5`, `sgdisk 1.0.10`,
|
||
`partprobe 3.6`, `mkfs.btrfs 6.14`, `mkfs.fat`.
|
||
|
||
Full install to a fresh 24 GiB qcow2 driven from the appliance:
|
||
|
||
- `INSTALL_RC=0`, all stages, `NO_SECRET_LEAK`.
|
||
|
||
Detached boot of the installed disk (ISO removed):
|
||
|
||
- systemd-boot -> LUKS2 unlock -> `phoebus` login;
|
||
- `systemctl is-system-running` -> `running`; `0 loaded units listed`;
|
||
- **Gentoo-only**: `Gentoo Base System release 2.18`, Portage 3.0.82.2,
|
||
`no-apt`; systemd PID 1;
|
||
- `cryptroot` Btrfs mounted at `/`, `/home`, `/var`, `/.snapshots`, `/swap`,
|
||
ESP `/efi`.
|
||
|
||
This is Bash-free and Gentoo-target-only: the appliance contributes tooling,
|
||
never configuration.
|
||
|
||
### Open items found in this run
|
||
|
||
- The target came up with the default `gentoo` hostname (empty
|
||
`/etc/hostname`) even though the plan/config specified `phoebus`, unlike the
|
||
earlier Gentoo-appliance installs. Needs a focused look (possible stage3 or
|
||
ordering difference).
|
||
- The first detached boot reported `enp0s2 DOWN`; DHCP timing was not
|
||
re-checked. The earlier Gentoo-appliance boot had `enp0s2 UP 10.0.2.15/24`.
|
||
|
||
Neither affects the proven Gentoo-appliance boot path; both are recorded rather
|
||
than hidden.
|
||
|
||
## Known remaining work
|
||
|
||
- The graphical session was verified with the **headless** wlroots backend
|
||
(no logind seat in the serial harness). A physical VT / seat path is
|
||
architected (`phoebus-session`) but was not exercised on real hardware.
|
||
- NVIDIA proprietary support is architecturally allowed but **NOT VERIFIED**.
|
||
- UKI is deferred (BLS is used).
|
||
- The installer rebuilds `sys-apps/systemd` from source because
|
||
boot/policykit/cryptsetup USE differ from the binhost. Shipping a matching
|
||
systemd binpkg is a future optimization.
|
||
- Desktop USE still does not match the signed binhost, so the desktop
|
||
transaction compiles a large amount from source (~26 minutes for the desktop
|
||
package set on 8 vCPUs in the test VM).
|