The region and keyboard fields were free-text, so users had to know zone names and layout names from memory, and the AI model had to be typed. Add an 'options' bridge op (live timezones/locales/keymaps) and an 'ai_catalog' op (the OpenCode provider/model catalog, or an OpenAI-compatible /models list), and a searchable PhoebusCombo picker. Also name provider/model in OpenCode errors instead of an opaque HTTP code. Record the user-trial findings U1-U5.
12 KiB
Verification and findings register
Companion to docs/STATUS.md. The machine-readable per-run results live outside
git under /var/tmp/phoebus-vm/<run-id>/results.json; this file is the
human-readable register of what was found, fixed and still unverified.
OpenCode adapter (2026-09-22, second pass)
Implemented a real OpenCode adapter (not just a probe), integrated through
canonical configuration (ai.backend = "opencode"), the GUI bridge, the
installer CLI and sys ask. Defects found while implementing and testing:
| ID | Defect | Fix | Test |
|---|---|---|---|
| O1 | No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport | phoebus-executor::opencode_client implements the real session/message protocol as a ModelTransport |
live + bridge end-to-end |
| O2 | POST /session was sent a model field per the OpenAPI schema |
the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message | real-HTTP integration |
| O3 | The probe had no credential input | ProbeRequest.auth + basic-auth header via stdin |
auth integration test |
| O4 | The probe did not enforce the connection destination | Endpoint::checked_target resolves and requires loopback for plain http; the connection is pinned |
transport_policy_enforces_loopback_for_plain_http, insecure-destination test |
| O5 | Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input | removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size | real-HTTP chunked/oversize/redirect/disconnect tests |
| O6 | The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog | status is checked before parsing; non-2xx is incompatible |
a_misleading_success_shaped_500_does_not_produce_a_catalog |
| O7 | Catalog presence counted as readiness | ready is reserved for a validated response; the probe returns model_catalog/reachable |
an_empty_catalog_is_reachable_not_ready |
| O8 | probe_live_server_when_configured passed when its env var was absent and only asserted reachability |
replaced by env-gated opencode_live tests that assert a real inference; PHOEBUS_OPENCODE_REQUIRE_LIVE=1 fails when the live run cannot happen |
make opencode-acceptance |
| O9 | The credential test built an already-sanitised struct | tests feed credential-bearing server bodies and assert errors/catalogs never contain them | one_shot_rejects_a_non_assistant_or_errored_reply, catalog_never_reads_provider_keys |
| O10 | A provider name satisfied the mandatory-AI gate | both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it | a_provider_name_alone_does_not_satisfy_the_ai_requirement, an_unreachable_provider_does_not_satisfy_the_ai_gate |
| O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly action + value with explicit examples |
live bridge ai_check |
| O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness |
| O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free phoebus-state lines; the harness asserts the current step and AI state |
harness |
| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in complete_proposal_at/complete_plan, with the strict parser still final |
a_rejected_proposal_is_retried_once_with_a_correction, a_retry_that_still_fails_is_still_rejected |
Graphical OpenCode acceptance (VM verified)
gui-driver.py opencode on the built ISO (real model, no simulation):
- welcome → installer → AI step; selected OpenCode agent through the UI;
- typed the server URL, provider and model through the UI;
- Test connection obtained a real model response (green "Model responded");
- Connect and validate obtained and validated a proposal (green "Validated",
frontend state
aiConnected=true); - run
gui-opencode-20260922T102640Z-be168813: 17/17 PASS, exit 0; the full GUI regressiongui-gui-20260922T102907Z-a9e5e824was 25/25 PASS. - The model is reached through a guest-loopback relay to the host OpenCode server (the SSH-tunnel shape), so the transport policy is satisfied.
- Authenticated server: the same harness with a password-protected server got
through authentication and a real Test connection; a flaky malformed proposal
was correctly rejected. Use
deepseek-flash(reliable) overdeepseek-v4-flash(occasionally malformed).
Still NOT RUN: the full graphical installation (disk → account → review → ERASE → install → detached boot) through OpenCode. The AI gate is now reached and passed in the graphical VM; the remaining stages are the long package installation. Two fresh-disk graphical installs and a separate physical machine remain unverified.
User trial findings (interactive VM, 2026-09-22)
Observed while a user drove the live ISO by hand (run
/var/tmp/phoebus-vm/user-session*):
| ID | Finding | Status |
|---|---|---|
| U1 | Timezone (and locale, keyboard) were free-text: the user was expected to know America/New_York etc. Only ~5 keymap presets were offered, then manual entry. |
Fixed: searchable pickers populated from the live system (418 timezones from zone.tab, 513 locales, 82 keymaps), type-to-filter and click-to-choose; a custom value is still accepted. |
| U2 | The AI provider/model fields were free-text; the user had to know the exact model id. | Fixed: ai_catalog bridge op returns the OpenCode catalog (223 providers, real model ids) or an OpenAI-compatible /models list; the GUI shows selectable Provider/Model combos with a "Load list" action. |
| U3 | An invalid model produced only OpenCode message failed: HTTP 500 — opaque. |
Fixed in source: the error now names provider/model and says to check the model exists and the credentials are valid. (Needs the next ISO.) |
| U4 | At first boot the DMS app-launcher/overview overlay was open on top of the welcome window, hiding the primary action until dismissed. | Open: reproduced once; investigating whether it is a DMS first-run behaviour or a focus race. |
| U5 | The OpenCode model default deepseek-chat is not present on every server, so the first attempt failed. |
Mitigated: the picker now lists only real models. Considering defaulting the model to unset and requiring a pick. |
None of these blocked the installer; they are usability defects. U1–U3 are implemented; U4–U5 are tracked.
OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP integration tests + 2 live tests (skipped offline, real inference when configured). Bridge gate tests: 5. Full suite: 336 Rust tests.
Run matrix (this session, 2026-09-22)
{
"source_revision": "7e9fa9c",
"iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
"static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
"runs": [
{"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
"artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
{"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
{"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
{"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
{"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
{"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
{"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
{"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
{"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
{"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
{"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
]
}
Confirmed defects fixed
| ID | Defect | Symptom | Fix | Regression test |
|---|---|---|---|---|
| D1 | Disk safety was not revalidated before destroying the target | A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review | Installer::revalidate_target_disk re-probes and compares identity in storage_prepare |
orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...} |
| D2 | probe_disks parsed human-readable lsblk sizes as bytes |
Every disk displayed 0 B; the size-identity check was blind |
pass -b to lsblk |
probe::tests::probe_disks_classifies_optical_as_protected asserts the byte size |
| D3 | The bridge could replay or overlap a destructive install | A second start_install (retry, second frontend, reconnect) could re-run storage preparation |
approval is single-use, consumed before work; installing/installed guards |
start_install_cannot_be_replayed_after_success |
| D4 | The welcome's primary action was not keyboard-focused | No focus ring; Return did not open the installer | request activation + forceActiveFocus(); regression test also rejects the invalid onActiveChanged signal |
welcome_primary_action_is_keyboard_reachable |
| D5 | Escape during installation silently closed the frontend | A destructive operation could be abandoned with no explanation | Escape is refused during progress and the UI says so | progress_cannot_be_abandoned_silently |
| D6 | Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist | A config-supplied endpoint could target file:, inject curl config, or return a huge body |
validate_endpoint, proto = "=http,https", max-filesize, redirects disabled |
endpoint_validation_rejects_unsafe_schemes_and_injection, curl_transport_refuses_an_unsafe_endpoint, curl_transport_bounds_the_response_and_disables_redirects |
| D7 | TargetRoot followed symlinks out of the target |
A planted symlink could redirect an installer write to the host | reject_symlink_components in write/ensure_dir |
target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf} |
| D8 | The VM harness documented disk safety but did not enforce it, and reported observations rather than results | A mistyped DISK could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion |
vmsafety.py; per-run OVMF/QMP; structured results; argument arrays |
test_vmsafety.py (16 tests), run by make test |
Suspected risks (reproduced or bounded)
- Prompt injection in device labels / model output. Typed proposals pass
deny_unknown_fieldsand a closed action set; device labels are display-only. Bounded by tests (phoebus-aiparse/allowlist). No VM canary run. - Live sudoers is
NOPASSWD: ALL(live-only). It does not reach the installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up. - OpenCode
/config/providersreturns provider credentials in cleartext. Upstream behaviour; the probe keeps only counts and docs mandate loopback + tunnel. Not a Phoebus defect, but a deployment hazard.
Missing verification (blocked / not run)
- Full graphical installation past the AI step and two fresh-disk graphical installs: BLOCKED — no provider key and no local model; normal install requires a connected model.
install/bootmodes on the final ISO: NOT RUN (the earlier appliance build passedINSTALL-RC=0+ detached boot; the final ISO's binary changed, so this is not automatically re-verified).- Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat, NVIDIA: NOT RUN / unverified.
- Second physical computer OpenCode transport: BLOCKED; host-local forwarded hop and unauthenticated-LAN refusal were tested instead.