phoebus_os/docs/VERIFICATION.md
Phoebus c16014ade8 feat(gui): select and validate OpenCode from the installer
Add an 'OpenCode agent' choice with server/password/provider/model fields,
a Test connection action, and a secret-free frontend state line the
harness asserts. Require exactly one installer session and assert the
current step and AI state instead of inferring from a screenshot. Add a
live-OpenCode acceptance make target and document the adapter, transport
policy, states and workflows (ADR 0011 updated).
2026-09-22 02:36:36 -07:00

8.9 KiB

Verification and findings register

Companion to docs/STATUS.md. The machine-readable per-run results live outside git under /var/tmp/phoebus-vm/<run-id>/results.json; this file is the human-readable register of what was found, fixed and still unverified.

OpenCode adapter (2026-09-22, second pass)

Implemented a real OpenCode adapter (not just a probe), integrated through canonical configuration (ai.backend = "opencode"), the GUI bridge, the installer CLI and sys ask. Defects found while implementing and testing:

ID Defect Fix Test
O1 No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport phoebus-executor::opencode_client implements the real session/message protocol as a ModelTransport live + bridge end-to-end
O2 POST /session was sent a model field per the OpenAPI schema the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message real-HTTP integration
O3 The probe had no credential input ProbeRequest.auth + basic-auth header via stdin auth integration test
O4 The probe did not enforce the connection destination Endpoint::checked_target resolves and requires loopback for plain http; the connection is pinned transport_policy_enforces_loopback_for_plain_http, insecure-destination test
O5 Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size real-HTTP chunked/oversize/redirect/disconnect tests
O6 The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog status is checked before parsing; non-2xx is incompatible a_misleading_success_shaped_500_does_not_produce_a_catalog
O7 Catalog presence counted as readiness ready is reserved for a validated response; the probe returns model_catalog/reachable an_empty_catalog_is_reachable_not_ready
O8 probe_live_server_when_configured passed when its env var was absent and only asserted reachability replaced by env-gated opencode_live tests that assert a real inference; PHOEBUS_OPENCODE_REQUIRE_LIVE=1 fails when the live run cannot happen make opencode-acceptance
O9 The credential test built an already-sanitised struct tests feed credential-bearing server bodies and assert errors/catalogs never contain them one_shot_rejects_a_non_assistant_or_errored_reply, catalog_never_reads_provider_keys
O10 A provider name satisfied the mandatory-AI gate both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it a_provider_name_alone_does_not_satisfy_the_ai_requirement, an_unreachable_provider_does_not_satisfy_the_ai_gate
O11 The proposal prompt allowed the model to key the change by action name prompt now requires exactly action + value with explicit examples live bridge ai_check
O12 The GUI duplicate-session assertion accepted two pre-existing sessions requires exactly one session before and after harness
O13 GUI navigation was inferred from a surviving process/screenshot the frontend emits secret-free phoebus-state lines; the harness asserts the current step and AI state harness

OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP integration tests + 2 live tests (skipped offline, real inference when configured). Bridge gate tests: 5. Full suite: 336 Rust tests.

Run matrix (this session, 2026-09-22)

{
  "source_revision": "7e9fa9c",
  "iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
  "static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
  "runs": [
    {"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
     "artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
    {"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
    {"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
    {"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
    {"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
    {"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
    {"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
    {"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
    {"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
    {"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
  ]
}

Confirmed defects fixed

ID Defect Symptom Fix Regression test
D1 Disk safety was not revalidated before destroying the target A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review Installer::revalidate_target_disk re-probes and compares identity in storage_prepare orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}
D2 probe_disks parsed human-readable lsblk sizes as bytes Every disk displayed 0 B; the size-identity check was blind pass -b to lsblk probe::tests::probe_disks_classifies_optical_as_protected asserts the byte size
D3 The bridge could replay or overlap a destructive install A second start_install (retry, second frontend, reconnect) could re-run storage preparation approval is single-use, consumed before work; installing/installed guards start_install_cannot_be_replayed_after_success
D4 The welcome's primary action was not keyboard-focused No focus ring; Return did not open the installer request activation + forceActiveFocus(); regression test also rejects the invalid onActiveChanged signal welcome_primary_action_is_keyboard_reachable
D5 Escape during installation silently closed the frontend A destructive operation could be abandoned with no explanation Escape is refused during progress and the UI says so progress_cannot_be_abandoned_silently
D6 Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist A config-supplied endpoint could target file:, inject curl config, or return a huge body validate_endpoint, proto = "=http,https", max-filesize, redirects disabled endpoint_validation_rejects_unsafe_schemes_and_injection, curl_transport_refuses_an_unsafe_endpoint, curl_transport_bounds_the_response_and_disables_redirects
D7 TargetRoot followed symlinks out of the target A planted symlink could redirect an installer write to the host reject_symlink_components in write/ensure_dir target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}
D8 The VM harness documented disk safety but did not enforce it, and reported observations rather than results A mistyped DISK could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion vmsafety.py; per-run OVMF/QMP; structured results; argument arrays test_vmsafety.py (16 tests), run by make test

Suspected risks (reproduced or bounded)

  • Prompt injection in device labels / model output. Typed proposals pass deny_unknown_fields and a closed action set; device labels are display-only. Bounded by tests (phoebus-ai parse/allowlist). No VM canary run.
  • Live sudoers is NOPASSWD: ALL (live-only). It does not reach the installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
  • OpenCode /config/providers returns provider credentials in cleartext. Upstream behaviour; the probe keeps only counts and docs mandate loopback + tunnel. Not a Phoebus defect, but a deployment hazard.

Missing verification (blocked / not run)

  • Full graphical installation past the AI step and two fresh-disk graphical installs: BLOCKED — no provider key and no local model; normal install requires a connected model.
  • install/boot modes on the final ISO: NOT RUN (the earlier appliance build passed INSTALL-RC=0 + detached boot; the final ISO's binary changed, so this is not automatically re-verified).
  • Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat, NVIDIA: NOT RUN / unverified.
  • Second physical computer OpenCode transport: BLOCKED; host-local forwarded hop and unauthenticated-LAN refusal were tested instead.