phoebus_os/docs/VERIFICATION.md
Phoebus cc23df69de test(vm): enforce harness disk safety and structured results
Validate every writable drive (qcow2, inside PHOEBUS_VM_DISKS, no /dev/*,
block/char devices, raw images, escaping symlinks or outside backing
chains), give each run its own OVMF variable file, QMP socket and state
directory, and use QMP plus argument arrays instead of shell interpolation.
Return nonzero on any failed assertion, timeout, EOF or missing result and
record PASS/FAIL/BLOCKED/SKIPPED/NOT RUN. Add 16 unit tests, a report
generator and a findings register; make test runs the harness suite.
2026-09-21 20:01:50 -07:00

5.6 KiB

Verification and findings register

Companion to docs/STATUS.md. The machine-readable per-run results live outside git under /var/tmp/phoebus-vm/<run-id>/results.json; this file is the human-readable register of what was found, fixed and still unverified.

Run matrix (this session, 2026-09-22)

{
  "source_revision": "7e9fa9c",
  "iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
  "static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
  "runs": [
    {"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
     "artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
    {"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
    {"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
    {"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
    {"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
    {"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
    {"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
    {"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
    {"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
  ]
}

Confirmed defects fixed

ID Defect Symptom Fix Regression test
D1 Disk safety was not revalidated before destroying the target A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review Installer::revalidate_target_disk re-probes and compares identity in storage_prepare orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}
D2 probe_disks parsed human-readable lsblk sizes as bytes Every disk displayed 0 B; the size-identity check was blind pass -b to lsblk probe::tests::probe_disks_classifies_optical_as_protected asserts the byte size
D3 The bridge could replay or overlap a destructive install A second start_install (retry, second frontend, reconnect) could re-run storage preparation approval is single-use, consumed before work; installing/installed guards start_install_cannot_be_replayed_after_success
D4 The welcome's primary action was not keyboard-focused No focus ring; Return did not open the installer request activation + forceActiveFocus(); regression test also rejects the invalid onActiveChanged signal welcome_primary_action_is_keyboard_reachable
D5 Escape during installation silently closed the frontend A destructive operation could be abandoned with no explanation Escape is refused during progress and the UI says so progress_cannot_be_abandoned_silently
D6 Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist A config-supplied endpoint could target file:, inject curl config, or return a huge body validate_endpoint, proto = "=http,https", max-filesize, redirects disabled endpoint_validation_rejects_unsafe_schemes_and_injection, curl_transport_refuses_an_unsafe_endpoint, curl_transport_bounds_the_response_and_disables_redirects
D7 TargetRoot followed symlinks out of the target A planted symlink could redirect an installer write to the host reject_symlink_components in write/ensure_dir target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}
D8 The VM harness documented disk safety but did not enforce it, and reported observations rather than results A mistyped DISK could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion vmsafety.py; per-run OVMF/QMP; structured results; argument arrays test_vmsafety.py (16 tests), run by make test

Suspected risks (reproduced or bounded)

  • Prompt injection in device labels / model output. Typed proposals pass deny_unknown_fields and a closed action set; device labels are display-only. Bounded by tests (phoebus-ai parse/allowlist). No VM canary run.
  • Live sudoers is NOPASSWD: ALL (live-only). It does not reach the installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
  • OpenCode /config/providers returns provider credentials in cleartext. Upstream behaviour; the probe keeps only counts and docs mandate loopback + tunnel. Not a Phoebus defect, but a deployment hazard.

Missing verification (blocked / not run)

  • Full graphical installation past the AI step and two fresh-disk graphical installs: BLOCKED — no provider key and no local model; normal install requires a connected model.
  • install/boot modes on the final ISO: NOT RUN (the earlier appliance build passed INSTALL-RC=0 + detached boot; the final ISO's binary changed, so this is not automatically re-verified).
  • Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat, NVIDIA: NOT RUN / unverified.
  • Second physical computer OpenCode transport: BLOCKED; host-local forwarded hop and unauthenticated-LAN refusal were tested instead.