phoebus_os/docs/VERIFICATION.md
Phoebus 359f083bc2
Some checks are pending
ci / test (push) Waiting to run
test(gui): drive the OpenCode AI step through the real UI
Add an 'opencode' harness mode with a guest-loopback relay (the tunnel
shape), keyboard selection of the OpenCode chip, UI entry of the server/
provider/model (and a 0600 server password), and assertions on the
frontend's secret-free state (tested, validated, applied). Record the VM
evidence and the remaining not-run full-install gate.
2026-09-22 03:32:07 -07:00

10 KiB

Verification and findings register

Companion to docs/STATUS.md. The machine-readable per-run results live outside git under /var/tmp/phoebus-vm/<run-id>/results.json; this file is the human-readable register of what was found, fixed and still unverified.

OpenCode adapter (2026-09-22, second pass)

Implemented a real OpenCode adapter (not just a probe), integrated through canonical configuration (ai.backend = "opencode"), the GUI bridge, the installer CLI and sys ask. Defects found while implementing and testing:

ID Defect Fix Test
O1 No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport phoebus-executor::opencode_client implements the real session/message protocol as a ModelTransport live + bridge end-to-end
O2 POST /session was sent a model field per the OpenAPI schema the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message real-HTTP integration
O3 The probe had no credential input ProbeRequest.auth + basic-auth header via stdin auth integration test
O4 The probe did not enforce the connection destination Endpoint::checked_target resolves and requires loopback for plain http; the connection is pinned transport_policy_enforces_loopback_for_plain_http, insecure-destination test
O5 Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size real-HTTP chunked/oversize/redirect/disconnect tests
O6 The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog status is checked before parsing; non-2xx is incompatible a_misleading_success_shaped_500_does_not_produce_a_catalog
O7 Catalog presence counted as readiness ready is reserved for a validated response; the probe returns model_catalog/reachable an_empty_catalog_is_reachable_not_ready
O8 probe_live_server_when_configured passed when its env var was absent and only asserted reachability replaced by env-gated opencode_live tests that assert a real inference; PHOEBUS_OPENCODE_REQUIRE_LIVE=1 fails when the live run cannot happen make opencode-acceptance
O9 The credential test built an already-sanitised struct tests feed credential-bearing server bodies and assert errors/catalogs never contain them one_shot_rejects_a_non_assistant_or_errored_reply, catalog_never_reads_provider_keys
O10 A provider name satisfied the mandatory-AI gate both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it a_provider_name_alone_does_not_satisfy_the_ai_requirement, an_unreachable_provider_does_not_satisfy_the_ai_gate
O11 The proposal prompt allowed the model to key the change by action name prompt now requires exactly action + value with explicit examples live bridge ai_check
O12 The GUI duplicate-session assertion accepted two pre-existing sessions requires exactly one session before and after harness
O13 GUI navigation was inferred from a surviving process/screenshot the frontend emits secret-free phoebus-state lines; the harness asserts the current step and AI state harness
O14 A model could return a malformed proposal with no second chance one bounded corrective retry in complete_proposal_at/complete_plan, with the strict parser still final a_rejected_proposal_is_retried_once_with_a_correction, a_retry_that_still_fails_is_still_rejected

Graphical OpenCode acceptance (VM verified)

gui-driver.py opencode on the built ISO (real model, no simulation):

  • welcome → installer → AI step; selected OpenCode agent through the UI;
  • typed the server URL, provider and model through the UI;
  • Test connection obtained a real model response (green "Model responded");
  • Connect and validate obtained and validated a proposal (green "Validated", frontend state aiConnected=true);
  • run gui-opencode-20260922T102640Z-be168813: 17/17 PASS, exit 0; the full GUI regression gui-gui-20260922T102907Z-a9e5e824 was 25/25 PASS.
  • The model is reached through a guest-loopback relay to the host OpenCode server (the SSH-tunnel shape), so the transport policy is satisfied.
  • Authenticated server: the same harness with a password-protected server got through authentication and a real Test connection; a flaky malformed proposal was correctly rejected. Use deepseek-flash (reliable) over deepseek-v4-flash (occasionally malformed).

Still NOT RUN: the full graphical installation (disk → account → review → ERASE → install → detached boot) through OpenCode. The AI gate is now reached and passed in the graphical VM; the remaining stages are the long package installation. Two fresh-disk graphical installs and a separate physical machine remain unverified.

OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP integration tests + 2 live tests (skipped offline, real inference when configured). Bridge gate tests: 5. Full suite: 336 Rust tests.

Run matrix (this session, 2026-09-22)

{
  "source_revision": "7e9fa9c",
  "iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
  "static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
  "runs": [
    {"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
     "artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
    {"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
    {"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
    {"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
    {"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
    {"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
    {"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
    {"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
    {"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
  ]
}

Confirmed defects fixed

ID Defect Symptom Fix Regression test
D1 Disk safety was not revalidated before destroying the target A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review Installer::revalidate_target_disk re-probes and compares identity in storage_prepare orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}
D2 probe_disks parsed human-readable lsblk sizes as bytes Every disk displayed 0 B; the size-identity check was blind pass -b to lsblk probe::tests::probe_disks_classifies_optical_as_protected asserts the byte size
D3 The bridge could replay or overlap a destructive install A second start_install (retry, second frontend, reconnect) could re-run storage preparation approval is single-use, consumed before work; installing/installed guards start_install_cannot_be_replayed_after_success
D4 The welcome's primary action was not keyboard-focused No focus ring; Return did not open the installer request activation + forceActiveFocus(); regression test also rejects the invalid onActiveChanged signal welcome_primary_action_is_keyboard_reachable
D5 Escape during installation silently closed the frontend A destructive operation could be abandoned with no explanation Escape is refused during progress and the UI says so progress_cannot_be_abandoned_silently
D6 Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist A config-supplied endpoint could target file:, inject curl config, or return a huge body validate_endpoint, proto = "=http,https", max-filesize, redirects disabled endpoint_validation_rejects_unsafe_schemes_and_injection, curl_transport_refuses_an_unsafe_endpoint, curl_transport_bounds_the_response_and_disables_redirects
D7 TargetRoot followed symlinks out of the target A planted symlink could redirect an installer write to the host reject_symlink_components in write/ensure_dir target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}
D8 The VM harness documented disk safety but did not enforce it, and reported observations rather than results A mistyped DISK could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion vmsafety.py; per-run OVMF/QMP; structured results; argument arrays test_vmsafety.py (16 tests), run by make test

Suspected risks (reproduced or bounded)

  • Prompt injection in device labels / model output. Typed proposals pass deny_unknown_fields and a closed action set; device labels are display-only. Bounded by tests (phoebus-ai parse/allowlist). No VM canary run.
  • Live sudoers is NOPASSWD: ALL (live-only). It does not reach the installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
  • OpenCode /config/providers returns provider credentials in cleartext. Upstream behaviour; the probe keeps only counts and docs mandate loopback + tunnel. Not a Phoebus defect, but a deployment hazard.

Missing verification (blocked / not run)

  • Full graphical installation past the AI step and two fresh-disk graphical installs: BLOCKED — no provider key and no local model; normal install requires a connected model.
  • install/boot modes on the final ISO: NOT RUN (the earlier appliance build passed INSTALL-RC=0 + detached boot; the final ISO's binary changed, so this is not automatically re-verified).
  • Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat, NVIDIA: NOT RUN / unverified.
  • Second physical computer OpenCode transport: BLOCKED; host-local forwarded hop and unauthenticated-LAN refusal were tested instead.