phoebus_os/docs/VERIFICATION.md
Phoebus 1106bcaa0a
Some checks are pending
ci / test (push) Waiting to run
docs(verification): record the graphical journey results and host-time blocker
Update the changelog, status, handoff, verification register and VM boot
record with the 2026-09-22 graphical journey work: the fixed step sequence,
the interaction-defect fixes (readiness refresh, catalog sync/staleness, AI
revalidation, keyboard suggestion navigation, truthful keyboard-step label,
link vs service readiness, busy/failure handling), the new gui-install
harness mode, and the final-ISO results.

The gui (28/28) and opencode (18/18) journeys pass; gui-install passes every
journey assertion and starts a real install, but the four-hour harness timeout
fired during the kernel build (base packages ~116 min), so two completed
fresh-disk graphical installs + detached boot remain BLOCKED on host build
time, not on a code defect.
2026-09-22 12:30:55 -07:00

16 KiB
Raw Blame History

Verification and findings register

Companion to docs/STATUS.md. The machine-readable per-run results live outside git under /var/tmp/phoebus-vm/<run-id>/results.json; this file is the human-readable register of what was found, fixed and still unverified.

OpenCode adapter (2026-09-22, second pass)

Implemented a real OpenCode adapter (not just a probe), integrated through canonical configuration (ai.backend = "opencode"), the GUI bridge, the installer CLI and sys ask. Defects found while implementing and testing:

ID Defect Fix Test
O1 No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport phoebus-executor::opencode_client implements the real session/message protocol as a ModelTransport live + bridge end-to-end
O2 POST /session was sent a model field per the OpenAPI schema the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message real-HTTP integration
O3 The probe had no credential input ProbeRequest.auth + basic-auth header via stdin auth integration test
O4 The probe did not enforce the connection destination Endpoint::checked_target resolves and requires loopback for plain http; the connection is pinned transport_policy_enforces_loopback_for_plain_http, insecure-destination test
O5 Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size real-HTTP chunked/oversize/redirect/disconnect tests
O6 The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog status is checked before parsing; non-2xx is incompatible a_misleading_success_shaped_500_does_not_produce_a_catalog
O7 Catalog presence counted as readiness ready is reserved for a validated response; the probe returns model_catalog/reachable an_empty_catalog_is_reachable_not_ready
O8 probe_live_server_when_configured passed when its env var was absent and only asserted reachability replaced by env-gated opencode_live tests that assert a real inference; PHOEBUS_OPENCODE_REQUIRE_LIVE=1 fails when the live run cannot happen make opencode-acceptance
O9 The credential test built an already-sanitised struct tests feed credential-bearing server bodies and assert errors/catalogs never contain them one_shot_rejects_a_non_assistant_or_errored_reply, catalog_never_reads_provider_keys
O10 A provider name satisfied the mandatory-AI gate both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it a_provider_name_alone_does_not_satisfy_the_ai_requirement, an_unreachable_provider_does_not_satisfy_the_ai_gate
O11 The proposal prompt allowed the model to key the change by action name prompt now requires exactly action + value with explicit examples live bridge ai_check
O12 The GUI duplicate-session assertion accepted two pre-existing sessions requires exactly one session before and after harness
O13 GUI navigation was inferred from a surviving process/screenshot the frontend emits secret-free phoebus-state lines; the harness asserts the current step and AI state harness
O14 A model could return a malformed proposal with no second chance one bounded corrective retry in complete_proposal_at/complete_plan, with the strict parser still final a_rejected_proposal_is_retried_once_with_a_correction, a_retry_that_still_fails_is_still_rejected

Graphical OpenCode acceptance (VM verified)

gui-driver.py opencode on the built ISO (real model, no simulation):

  • welcome → installer → AI step; selected OpenCode agent through the UI;
  • typed the server URL, provider and model through the UI;
  • Test connection obtained a real model response (green "Model responded");
  • Connect and validate obtained and validated a proposal (green "Validated", frontend state aiConnected=true);
  • run gui-opencode-20260922T192457Z-01714c92: 18/18 PASS, exit 0; the full GUI regression gui-gui-20260922T191954Z-876d5e91 was 28/28 PASS (both on the final ISO sha256 0c386a1b…).
  • The model is reached through a guest-loopback relay to the host OpenCode server (the SSH-tunnel shape), so the transport policy is satisfied.
  • Authenticated server: the same harness with a password-protected server got through authentication and a real Test connection; a flaky malformed proposal was correctly rejected. Use deepseek-flash (reliable) over deepseek-v4-flash (occasionally malformed).

Graphical journey: interaction defects fixed (2026-09-22)

ID Defect Fix Test
G1 Network Re-check ran the handshake init path and reset the step, clearing the selected disk a read-only readiness bridge op refreshes the probe without touching the form; only the first handshake initialises harness readiness-refresh-preserves-wizard, disk-survives-backtrack; Rust readiness_reports_link_and_service_reachability_separately
G2 Model-catalog requests used stale configuration/credentials, and a stale reply could overwrite a newer one the catalog syncs edited config + secrets first, replies carry a stable error_kind (auth required/rejected, insecure transport, model unavailable, unreachable), and a request sequence ignores superseded replies Rust ai_catalog_errors_are_machine_readable, connection_error_kind_classifies_common_failures; harness exercises the auth error
G3 The green AI indicators survived edits to provider/model/endpoint/credentials every connection-field edit invalidates the validated state; account/disk edits preserve it harness ai-invalidated-on-model-edit; Rust a_config_change_invalidates_the_validated_interaction
G4 Suggestion lists were mouse-only Up/Down move a visible highlight, Enter selects, Escape dismisses, Accessible roles; Enter in an open list is consumed harness keyboard-suggestion-select
G5 Selecting a keyboard layout implied it applied to the live session the step states the live session keeps its layout and the choice applies to the installed system manual review of the language step
G6 Readiness only detected a default route link and verified services reachability are reported separately Rust readiness_reports_link_and_service_reachability_separately
G7 A bridge crash left requests/busy indicators hanging the bridge fails every outstanding request on exit; single-flight guards and cleared busy states with retry Rust bridge exit path; harness frontend-no-bridge-errors
G8 PhoebusToggle Return/Enter called a signal handler as a function a single Keys.onPressed handler toggles on Space/Return/Enter harness gui-install-erase-confirmed

Full graphical install

gui-driver.py gui-install drives the real controls on a fresh 30 GiB qcow2 (run gui-gui-install-20260922T151232Z-0ef19969): 24 PASS / 1 FAIL. Every journey assertion passed (OpenCode connect/review/accept, disk, account, ERASE, confirmation toggle) and the backend reported the real stage sequence with elapsed time and no fabricated percentage. The one FAIL is gui-install-completes: the four-hour harness timeout fired during kernel_install (base packages ~116 min; the AI-selected optimized kernel compiled for >100 min on this host). The completed install + detached boot gate is BLOCKED on host build time; the deterministic noninteractive install + detached boot remains the proven boot path.

User trial findings (interactive VM, 2026-09-22)

Observed while a user drove the live ISO by hand (run /var/tmp/phoebus-vm/user-session*):

ID Finding Status
U1 Timezone (and locale, keyboard) were free-text: the user was expected to know America/New_York etc. Only ~5 keymap presets were offered, then manual entry. Fixed: searchable pickers populated from the live system (418 timezones from zone.tab, 513 locales, 82 keymaps), type-to-filter and click-to-choose; a custom value is still accepted.
U2 The AI provider/model fields were free-text; the user had to know the exact model id. Fixed: ai_catalog bridge op returns the OpenCode catalog (223 providers, real model ids) or an OpenAI-compatible /models list; the GUI shows selectable Provider/Model combos with a "Load list" action.
U3 An invalid model produced only OpenCode message failed: HTTP 500 — opaque. Fixed in source: the error now names provider/model and says to check the model exists and the credentials are valid. (Needs the next ISO.)
U4 At first boot the DMS app-launcher/overview overlay was open on top of the welcome window, hiding the primary action until dismissed. Open: reproduced once; investigating whether it is a DMS first-run behaviour or a focus race.
U5 The OpenCode model default deepseek-chat is not present on every server, so the first attempt failed. Mitigated: the picker now lists only real models. Considering defaulting the model to unset and requiring a pick.
U6 Showstopper. A fully-validated OpenCode install was refused at the final step with "installing Phoebus requires a successful, validated AI interaction". The validated state was cleared by unrelated configuration and secret changes (selecting a disk, typing the account password, setting the LUKS passphrase). Fixed: the AI validation is now bound to a digest of only the AI connection settings; non-AI config changes preserve it, and only an AI provider/model/endpoint/server change or an AI credential change invalidates it. Regression tests: a_non_ai_config_change_keeps_the_validated_interaction, an_account_secret_does_not_invalidate_but_an_ai_key_does.
U7 The account/encryption step required several attempts; the password/passphrase minimum was only revealed as a red error after pressing Next, and the offending fields could be scrolled off-screen. Mitigated: password/passphrase fields now carry an "At least 8 characters" / "Repeat …" placeholder hint before submission.

U6 blocked a real install; U1–U3 and U6–U7 are implemented, U4 remains open. The user's session reached the AI step, obtained and applied a validated OpenCode proposal, selected the disk, completed the account step and the review — and only then hit U6. Evidence: /var/tmp/phoebus-vm/user-session3/ (shots/0008–0016, observations.log).

OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP integration tests + 2 live tests (skipped offline, real inference when configured). Bridge gate tests: 5. Full suite: 336 Rust tests.

Run matrix (this session, 2026-09-22)

{
  "source_revision": "7e9fa9c",
  "iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
  "static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
  "runs": [
    {"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
     "artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
    {"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
    {"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
    {"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
    {"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
    {"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
    {"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
    {"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
    {"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
  ]
}

Confirmed defects fixed

ID Defect Symptom Fix Regression test
D1 Disk safety was not revalidated before destroying the target A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review Installer::revalidate_target_disk re-probes and compares identity in storage_prepare orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}
D2 probe_disks parsed human-readable lsblk sizes as bytes Every disk displayed 0 B; the size-identity check was blind pass -b to lsblk probe::tests::probe_disks_classifies_optical_as_protected asserts the byte size
D3 The bridge could replay or overlap a destructive install A second start_install (retry, second frontend, reconnect) could re-run storage preparation approval is single-use, consumed before work; installing/installed guards start_install_cannot_be_replayed_after_success
D4 The welcome's primary action was not keyboard-focused No focus ring; Return did not open the installer request activation + forceActiveFocus(); regression test also rejects the invalid onActiveChanged signal welcome_primary_action_is_keyboard_reachable
D5 Escape during installation silently closed the frontend A destructive operation could be abandoned with no explanation Escape is refused during progress and the UI says so progress_cannot_be_abandoned_silently
D6 Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist A config-supplied endpoint could target file:, inject curl config, or return a huge body validate_endpoint, proto = "=http,https", max-filesize, redirects disabled endpoint_validation_rejects_unsafe_schemes_and_injection, curl_transport_refuses_an_unsafe_endpoint, curl_transport_bounds_the_response_and_disables_redirects
D7 TargetRoot followed symlinks out of the target A planted symlink could redirect an installer write to the host reject_symlink_components in write/ensure_dir target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}
D8 The VM harness documented disk safety but did not enforce it, and reported observations rather than results A mistyped DISK could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion vmsafety.py; per-run OVMF/QMP; structured results; argument arrays test_vmsafety.py (16 tests), run by make test

Suspected risks (reproduced or bounded)

  • Prompt injection in device labels / model output. Typed proposals pass deny_unknown_fields and a closed action set; device labels are display-only. Bounded by tests (phoebus-ai parse/allowlist). No VM canary run.
  • Live sudoers is NOPASSWD: ALL (live-only). It does not reach the installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
  • OpenCode /config/providers returns provider credentials in cleartext. Upstream behaviour; the probe keeps only counts and docs mandate loopback + tunnel. Not a Phoebus defect, but a deployment hazard.

Missing verification (blocked / not run)

  • Full graphical installation past the AI step and two fresh-disk graphical installs: BLOCKED — no provider key and no local model; normal install requires a connected model.
  • install/boot modes on the final ISO: NOT RUN (the earlier appliance build passed INSTALL-RC=0 + detached boot; the final ISO's binary changed, so this is not automatically re-verified).
  • Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat, NVIDIA: NOT RUN / unverified.
  • Second physical computer OpenCode transport: BLOCKED; host-local forwarded hop and unauthenticated-LAN refusal were tested instead.