docs(evidence): record the iterative VM sys verification loop
Some checks are pending
ci / test (push) Waiting to run

This commit is contained in:
Phoebus 2026-09-29 03:06:09 -07:00
parent 1d982a3760
commit 685321f85b
4 changed files with 62 additions and 4 deletions

View file

@ -23,6 +23,13 @@ this file is the curated, human-readable summary.
`AI_ROOT_RC=0` (system sandbox), `AI_USER_RC=0` (`--user` sandbox, was `AI_ROOT_RC=0` (system sandbox), `AI_USER_RC=0` (`--user` sandbox, was
RC=124 timeout), `AI_NOSANDBOX_RC=0`, hostile `run_shell` reply `AI_SHELL_RC=1`; RC=124 timeout), `AI_NOSANDBOX_RC=0`, hostile `run_shell` reply `AI_SHELL_RC=1`;
the expanded `sys-acceptance.sh` matrix reports `SUMMARY PASS=49 FAIL=0`. the expanded `sys-acceptance.sh` matrix reports `SUMMARY PASS=49 FAIL=0`.
Additional loops on the same golden overlay pass: mutating acceptance as root
`SUMMARY PASS=42 FAIL=0` (`mutate-install`/`mutate-remove`/`mutate-history`),
rollback acceptance `PASS=11` (snapshot → change → `sys undo` → detached
verify), and the Snap journey `PASS=11`
(`serial-boot-20260929T093329Z-4ef893cf`,
`serial-rollback-20260929T094640Z-256c2ba1`,
`serial-boot-20260929T094957Z-3d4a9211`).
Also observed: a read-only `flatpak search` can trigger Flatpak's interactive Also observed: a read-only `flatpak search` can trigger Flatpak's interactive
`appstream-update` polkit prompt when the session is not recognized as active `appstream-update` polkit prompt when the session is not recognized as active
(headless serial), which can block an unattended run; on a real active seat (headless serial), which can block an unattended run; on a real active seat

View file

@ -40,8 +40,13 @@ commit it. It should be short and current; use `git log` for detail.
`sys-acceptance.sh` `SUMMARY PASS=49 FAIL=0`. Hypothesis recorded (not `sys-acceptance.sh` `SUMMARY PASS=49 FAIL=0`. Hypothesis recorded (not
changed): headless/inactive-session `flatpak search` can trigger Flatpak's changed): headless/inactive-session `flatpak search` can trigger Flatpak's
interactive `appstream-update` polkit prompt; the unattended acceptance runs interactive `appstream-update` polkit prompt; the unattended acceptance runs
detached to avoid the hang. Next: mutate acceptance, rollback, Snap, source detached to avoid the hang. Additional loops on the same golden overlay with
conversion / tuned kernel on the same golden overlay. the fixed binary: mutating acceptance as root `SUMMARY PASS=42 FAIL=0`
(`serial-boot-20260929T093329Z-4ef893cf`), rollback `PASS=11`
(`serial-rollback-20260929T094640Z-256c2ba1`), Snap journey `PASS=11`
(`serial-boot-20260929T094957Z-3d4a9211`). Next: source conversion / tuned
kernel (multi-hour compiles) and a live direct-provider model in the VM remain
NotRun.
- **Boundary-hardening pass (2026-09-28, IMPLEMENTED + UNIT-TESTED, VM NOT - **Boundary-hardening pass (2026-09-28, IMPLEMENTED + UNIT-TESTED, VM NOT
RUN):** six host-provable defects fixed with regression tests. (1) Removed RUN):** six host-provable defects fixed with regression tests. (1) Removed
the `PHOEBUS_ALLOW_UNPRIVILEGED_EXEC=1` env backdoor: it made the `PHOEBUS_ALLOW_UNPRIVILEGED_EXEC=1` env backdoor: it made

View file

@ -112,11 +112,11 @@ Booting that recovery entry offline remains a manual gate.
| Personal preferences + alert state (user-owned, system-default precedence) | yes | yes (save/reload/isolation/precedence/failure) | yes (installed VM: saved to user config, exit 0) | no | | Personal preferences + alert state (user-owned, system-default precedence) | yes | yes (save/reload/isolation/precedence/failure) | yes (installed VM: saved to user config, exit 0) | no |
| Personal write ownership (elevated chown, symlink refusal, cross-uid refusal) | yes | yes (ownership-fixer seam + CLI refusal) | no | no | | Personal write ownership (elevated chown, symlink refusal, cross-uid refusal) | yes | yes (ownership-fixer seam + CLI refusal) | no | no |
| Personal preference does not need the root-owned system transaction log | yes | yes (CLI regression: read-only transaction dir) | yes (installed VM acceptance) | n/a | | Personal preference does not need the root-owned system transaction log | yes | yes (CLI regression: read-only transaction dir) | yes (installed VM acceptance) | n/a |
| Installed `sys` acceptance matrix (read-only/failure) | yes | yes | yes (installed VM: `PASS=40 FAIL=0`, catalogue + regression group; **plus service-lifecycle, versioned-atom and path-normalization cases added 2026-09-28 — host-level `tests/sys_flaws.rs` 9/9 PASS, VM rerun pending**) | no | | Installed `sys` acceptance matrix (read-only/failure) | yes | yes | yes (installed VM, HEAD binary on a golden overlay `serial-boot-20260929T092147Z-6f88907a`: read-only `SUMMARY PASS=49 FAIL=0`; root mutate `serial-boot-20260929T093329Z-4ef893cf`: `SUMMARY PASS=42 FAIL=0` incl. `mutate-install`/`mutate-remove`/`mutate-history`) | no |
| `sys` adversarial input battery (injection/traversal/boundaries) | yes | yes (54-case `tests/sys_adversarial.rs`; five validation gaps fixed: file-path metacharacters + `..`, application-name traversal, ebuild `..` category (real overlay escape), Flatpak empty element, Nix option prefix) | no | no | | `sys` adversarial input battery (injection/traversal/boundaries) | yes | yes (54-case `tests/sys_adversarial.rs`; five validation gaps fixed: file-path metacharacters + `..`, application-name traversal, ebuild `..` category (real overlay escape), Flatpak empty element, Nix option prefix) | no | no |
| Portage config readable for non-root queries (`normalize_config_permissions`) | yes | yes (modes + keyring exclusion) | yes (fresh VM install: 0755/0644 out of the box; non-root `search` returns 3 results, `updates` reports 10) | no | | Portage config readable for non-root queries (`normalize_config_permissions`) | yes | yes (modes + keyring exclusion) | yes (fresh VM install: 0755/0644 out of the box; non-root `search` returns 3 results, `updates` reports 10) | no |
| Privileged catalogue mutations (install/remove/depclean/cache/apply updates, alternate-root) | yes | yes (unit) | yes (fresh installed VM via root: 7/7 PASS) | no | | Privileged catalogue mutations (install/remove/depclean/cache/apply updates, alternate-root) | yes | yes (unit) | yes (fresh installed VM via root: 7/7 PASS) | no |
| Rollback acceptance (snapshot -> change -> `sys undo` -> detached verify) | yes | yes | yes (`serial-rollback-20260926T030237Z-912f6386` PASS=11 FAIL=0 on a fresh disposable clone: package DB changed then restored, root marker gone, `/home` survives, boot running, no secret leak) | no | | Rollback acceptance (snapshot -> change -> `sys undo` -> detached verify) | yes | yes | yes (`serial-rollback-20260926T030237Z-912f6386` PASS=11 FAIL=0; re-verified with the HEAD binary `serial-rollback-20260929T094640Z-256c2ba1` PASS=11 on a fresh disposable overlay: package DB changed then restored, root marker gone, `/home` survives, systemd running, recovery entry present) | no |
| Privileged `--broker` plan execution + replay rejection in a VM (pkexec path) | yes | yes (privileged entry point) | yes (non-root `pkexec` with a terminal polkit agent: cancelled auth, broker JSON success, replay rejected; required the shipped policy + wheel-admin rule) | n/a | | Privileged `--broker` plan execution + replay rejection in a VM (pkexec path) | yes | yes (privileged entry point) | yes (non-root `pkexec` with a terminal polkit agent: cancelled auth, broker JSON success, replay rejected; required the shipped policy + wheel-admin rule) | n/a |
| Cache-cleanup preview is non-destructive (real `eclean`/`nix-store`) | yes | yes (preview uses `--pretend --deep` / `--print-dead`; sorted bound items) | yes (real `eclean`: upstream preview, cancelled approval, approved run, drift refusal, failed cleaner; user data preserved) | n/a | | Cache-cleanup preview is non-destructive (real `eclean`/`nix-store`) | yes | yes (preview uses `--pretend --deep` / `--print-dead`; sorted bound items) | yes (real `eclean`: upstream preview, cancelled approval, approved run, drift refusal, failed cleaner; user data preserved) | n/a |
| Reverse-dependency inspection includes installed targets | yes | yes (self-reference compares the dependent, not the target) | yes (real `sys-libs/glibc`, `sys-libs/zlib` dependents) | n/a | | Reverse-dependency inspection includes installed targets | yes | yes (self-reference compares the dependent, not the target) | yes (real `sys-libs/glibc`, `sys-libs/zlib` dependents) | n/a |

View file

@ -3,6 +3,52 @@
This records the first successful end-to-end install and boot of Phoebus OS This records the first successful end-to-end install and boot of Phoebus OS
against a disposable qcow2 disk. No physical disk was ever touched. against a disposable qcow2 disk. No physical disk was ever touched.
## 2026-09-29 Iterative `sys` VM loop on the golden overlay (AI sandbox + journeys)
Fresh qcow2 overlays were created on the golden installed image
(`/var/tmp/phoebus-vm/disks/phoebus-headless-80g.qcow2`) and booted detached with
the static `sys` built from HEAD; the base image was never modified.
- **Loop 1 (`serial-boot-20260929T074650Z-2f8fff0b`) found two real defects:**
(a) unprivileged `sys ask` hung on a polkit
`org.freedesktop.systemd1.manage-units` password prompt because the model
sandbox used the system manager (`RC=124` after a 200 s timeout); (b) an
unprivileged read-only AI plan printed the plan then exited 1 because the
service layer required the root-owned `/var/lib/phoebus/transactions.toml`.
- **Loop 2 (`serial-boot-20260929T075731Z-e4696546`) hit Flatpak's interactive
`appstream-update` polkit prompt** during a read-only `flatpak search` on the
headless serial session (the session is not recognized as polkit-active).
Polkit's upstream default is `allow_active=yes`, so a real active seat is not
affected; the unattended acceptance now runs detached so it cannot hang.
- **Fixes:** sandbox uses `systemd-run --user` with the user-manager property
subset for unprivileged callers (full system sandbox for root); transaction
recording is best-effort for non-root plans and mandatory for privileged
mutations.
- **Loop 3 (`serial-boot-20260929T084043Z-cfa7f7a5`):** read-only acceptance
reached `SUMMARY PASS=49 FAIL=0`; the root sandbox test timed out only because
the cached `sudo` timestamp expired during the long matrix (harness ordering).
- **Loop 4 (`serial-boot-20260929T092147Z-6f88907a`, driver PASS=8):**
system sandbox `AI_ROOT_RC=0`; user sandbox `AI_USER_RC=0` (was 124);
no-sandbox `AI_NOSANDBOX_RC=0`; hostile model `run_shell` reply
`AI_SHELL_RC=1`; read-only acceptance `SUMMARY PASS=49 FAIL=0`; 0 failed
units; clean poweroff.
- **Loop 5 (`serial-boot-20260929T093329Z-4ef893cf`, driver PASS=7):**
mutating acceptance as root `SUMMARY PASS=42 FAIL=0`, including
`mutate-install`, `mutate-remove`, `mutate-history` on `app-misc/hello`.
- **Loop 6 (`serial-rollback-20260929T094640Z-256c2ba1`, PASS=11):**
snapshot → change → `sys undo` → detached verify with the new binary
(`rollback-package-state-restored`, `rollback-home-survives`,
`rollback-root-restored`, `rollback-recovery-entry`, `rollback-status-recorded`).
- **Loop 7 (`serial-boot-20260929T094957Z-3d4a9211`, PASS=11):** Snap journey
(`snap-sys`, `snap-portage`, `snap-socket`, `snap-confinement`, `snap-install`,
`snap-refresh`, `snap-remove`); confinement remains `partial`, and a
revision-changing refresh / strict AppArmor confinement still needs an
external Store revision and an AppArmor-ready guest kernel.
- **NotRun in this loop:** source conversion and tuned-kernel build/boot/fallback
(multi-hour compiles on the disposable image); a live direct-provider model in
the VM (no credential/GPU); recovery-entry BLS selection (manual gate);
NVIDIA.
## 2026-09-28 Headless dual-target golden install (Run 13) & detached boot proven ## 2026-09-28 Headless dual-target golden install (Run 13) & detached boot proven
- **Golden install completed (Run 13, `serial-install-20260928T002557Z-e39d1934`):** - **Golden install completed (Run 13, `serial-install-20260928T002557Z-e39d1934`):**