phoebus_os/docs/VERIFICATION.md
Phoebus 1106bcaa0a
Some checks are pending
ci / test (push) Waiting to run
docs(verification): record the graphical journey results and host-time blocker
Update the changelog, status, handoff, verification register and VM boot
record with the 2026-09-22 graphical journey work: the fixed step sequence,
the interaction-defect fixes (readiness refresh, catalog sync/staleness, AI
revalidation, keyboard suggestion navigation, truthful keyboard-step label,
link vs service readiness, busy/failure handling), the new gui-install
harness mode, and the final-ISO results.

The gui (28/28) and opencode (18/18) journeys pass; gui-install passes every
journey assertion and starts a real install, but the four-hour harness timeout
fired during the kernel build (base packages ~116 min), so two completed
fresh-disk graphical installs + detached boot remain BLOCKED on host build
time, not on a code defect.
2026-09-22 12:30:55 -07:00

159 lines
16 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Verification and findings register
Companion to `docs/STATUS.md`. The machine-readable per-run results live outside
git under `/var/tmp/phoebus-vm/<run-id>/results.json`; this file is the
human-readable register of what was found, fixed and still unverified.
## OpenCode adapter (2026-09-22, second pass)
Implemented a real OpenCode adapter (not just a probe), integrated through
canonical configuration (`ai.backend = "opencode"`), the GUI bridge, the
installer CLI and `sys ask`. Defects found while implementing and testing:
| ID | Defect | Fix | Test |
|----|--------|-----|------|
| O1 | No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport | `phoebus-executor::opencode_client` implements the real session/message protocol as a `ModelTransport` | live + bridge end-to-end |
| O2 | `POST /session` was sent a `model` field per the OpenAPI schema | the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message | real-HTTP integration |
| O3 | The probe had no credential input | `ProbeRequest.auth` + basic-auth header via stdin | auth integration test |
| O4 | The probe did not enforce the connection destination | `Endpoint::checked_target` resolves and requires loopback for plain http; the connection is pinned | `transport_policy_enforces_loopback_for_plain_http`, insecure-destination test |
| O5 | Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input | removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size | real-HTTP chunked/oversize/redirect/disconnect tests |
| O6 | The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog | status is checked before parsing; non-2xx is `incompatible` | `a_misleading_success_shaped_500_does_not_produce_a_catalog` |
| O7 | Catalog presence counted as readiness | `ready` is reserved for a validated response; the probe returns `model_catalog`/`reachable` | `an_empty_catalog_is_reachable_not_ready` |
| O8 | `probe_live_server_when_configured` passed when its env var was absent and only asserted reachability | replaced by env-gated `opencode_live` tests that assert a real inference; `PHOEBUS_OPENCODE_REQUIRE_LIVE=1` fails when the live run cannot happen | `make opencode-acceptance` |
| O9 | The credential test built an already-sanitised struct | tests feed credential-bearing server bodies and assert errors/catalogs never contain them | `one_shot_rejects_a_non_assistant_or_errored_reply`, `catalog_never_reads_provider_keys` |
| O10 | A provider name satisfied the mandatory-AI gate | both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it | `a_provider_name_alone_does_not_satisfy_the_ai_requirement`, `an_unreachable_provider_does_not_satisfy_the_ai_gate` |
| O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` |
| O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness |
| O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness |
| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in `complete_proposal_at`/`complete_plan`, with the strict parser still final | `a_rejected_proposal_is_retried_once_with_a_correction`, `a_retry_that_still_fails_is_still_rejected` |
### Graphical OpenCode acceptance (VM verified)
`gui-driver.py opencode` on the built ISO (real model, no simulation):
- welcome → installer → AI step; selected **OpenCode agent** through the UI;
- typed the server URL, provider and model through the UI;
- **Test connection** obtained a real model response (green "Model responded");
- **Connect and validate** obtained and validated a proposal (green "Validated",
frontend state `aiConnected=true`);
- run `gui-opencode-20260922T192457Z-01714c92`: **18/18 PASS**, exit 0; the full
GUI regression `gui-gui-20260922T191954Z-876d5e91` was **28/28 PASS** (both on
the final ISO `sha256 0c386a1b…`).
- The model is reached through a guest-loopback relay to the host OpenCode
server (the SSH-tunnel shape), so the transport policy is satisfied.
- Authenticated server: the same harness with a password-protected server got
through authentication and a real Test connection; a flaky malformed proposal
was correctly rejected. Use `deepseek-flash` (reliable) over
`deepseek-v4-flash` (occasionally malformed).
### Graphical journey: interaction defects fixed (2026-09-22)
| ID | Defect | Fix | Test |
|----|--------|-----|------|
| G1 | Network **Re-check** ran the handshake init path and reset the step, clearing the selected disk | a read-only `readiness` bridge op refreshes the probe without touching the form; only the first handshake initialises | harness `readiness-refresh-preserves-wizard`, `disk-survives-backtrack`; Rust `readiness_reports_link_and_service_reachability_separately` |
| G2 | Model-catalog requests used stale configuration/credentials, and a stale reply could overwrite a newer one | the catalog syncs edited config + secrets first, replies carry a stable `error_kind` (auth required/rejected, insecure transport, model unavailable, unreachable), and a request sequence ignores superseded replies | Rust `ai_catalog_errors_are_machine_readable`, `connection_error_kind_classifies_common_failures`; harness exercises the auth error |
| G3 | The green AI indicators survived edits to provider/model/endpoint/credentials | every connection-field edit invalidates the validated state; account/disk edits preserve it | harness `ai-invalidated-on-model-edit`; Rust `a_config_change_invalidates_the_validated_interaction` |
| G4 | Suggestion lists were mouse-only | Up/Down move a visible highlight, Enter selects, Escape dismisses, `Accessible` roles; Enter in an open list is consumed | harness `keyboard-suggestion-select` |
| G5 | Selecting a keyboard layout implied it applied to the live session | the step states the live session keeps its layout and the choice applies to the installed system | manual review of the language step |
| G6 | Readiness only detected a default route | `link` and verified `services` reachability are reported separately | Rust `readiness_reports_link_and_service_reachability_separately` |
| G7 | A bridge crash left requests/busy indicators hanging | the bridge fails every outstanding request on exit; single-flight guards and cleared busy states with retry | Rust bridge exit path; harness `frontend-no-bridge-errors` |
| G8 | `PhoebusToggle` Return/Enter called a signal handler as a function | a single `Keys.onPressed` handler toggles on Space/Return/Enter | harness `gui-install-erase-confirmed` |
### Full graphical install
`gui-driver.py gui-install` drives the real controls on a fresh 30 GiB qcow2
(run `gui-gui-install-20260922T151232Z-0ef19969`): **24 PASS / 1 FAIL**. Every
journey assertion passed (OpenCode connect/review/accept, disk, account, ERASE,
confirmation toggle) and the backend reported the real stage sequence with
elapsed time and no fabricated percentage. The one FAIL is
`gui-install-completes`: the four-hour harness timeout fired during
`kernel_install` (base packages ~116 min; the AI-selected optimized kernel
compiled for >100 min on this host). The completed install + detached boot gate
is **BLOCKED on host build time**; the deterministic noninteractive install +
detached boot remains the proven boot path.
### User trial findings (interactive VM, 2026-09-22)
Observed while a user drove the live ISO by hand (run
`/var/tmp/phoebus-vm/user-session*`):
| ID | Finding | Status |
|----|---------|--------|
| U1 | Timezone (and locale, keyboard) were **free-text**: the user was expected to know `America/New_York` etc. Only ~5 keymap presets were offered, then manual entry. | **Fixed**: searchable pickers populated from the live system (418 timezones from `zone.tab`, 513 locales, 82 keymaps), type-to-filter and click-to-choose; a custom value is still accepted. |
| U2 | The AI **provider/model** fields were free-text; the user had to know the exact model id. | **Fixed**: `ai_catalog` bridge op returns the OpenCode catalog (223 providers, real model ids) or an OpenAI-compatible `/models` list; the GUI shows selectable Provider/Model combos with a "Load list" action. |
| U3 | An invalid model produced only `OpenCode message failed: HTTP 500` — opaque. | **Fixed in source**: the error now names `provider/model` and says to check the model exists and the credentials are valid. (Needs the next ISO.) |
| U4 | At first boot the DMS app-launcher/overview overlay was open on top of the welcome window, hiding the primary action until dismissed. | **Open**: reproduced once; investigating whether it is a DMS first-run behaviour or a focus race. |
| U5 | The OpenCode model default `deepseek-chat` is not present on every server, so the first attempt failed. | **Mitigated**: the picker now lists only real models. Considering defaulting the model to unset and requiring a pick. |
| U6 | **Showstopper.** A fully-validated OpenCode install was refused at the final step with *"installing Phoebus requires a successful, validated AI interaction"*. The validated state was cleared by **unrelated** configuration and secret changes (selecting a disk, typing the account password, setting the LUKS passphrase). | **Fixed**: the AI validation is now bound to a digest of only the AI connection settings; non-AI config changes preserve it, and only an AI provider/model/endpoint/server change or an AI credential change invalidates it. Regression tests: `a_non_ai_config_change_keeps_the_validated_interaction`, `an_account_secret_does_not_invalidate_but_an_ai_key_does`. |
| U7 | The account/encryption step required several attempts; the password/passphrase minimum was only revealed as a red error after pressing Next, and the offending fields could be scrolled off-screen. | **Mitigated**: password/passphrase fields now carry an "At least 8 characters" / "Repeat …" placeholder hint before submission. |
U6 blocked a real install; U1–U3 and U6–U7 are implemented, U4 remains open.
The user's session reached the AI step, obtained and applied a validated
OpenCode proposal, selected the disk, completed the account step and the
review — and only then hit U6. Evidence: `/var/tmp/phoebus-vm/user-session3/`
(`shots/0008`–`0016`, `observations.log`).
OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP
integration tests + 2 live tests (skipped offline, real inference when
configured). Bridge gate tests: 5. Full suite: 336 Rust tests.
## Run matrix (this session, 2026-09-22)
```json
{
"source_revision": "7e9fa9c",
"iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
"static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
"runs": [
{"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
"artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
{"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
{"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
{"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
{"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
{"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
{"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
{"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
{"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
{"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
{"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
]
}
```
## Confirmed defects fixed
| ID | Defect | Symptom | Fix | Regression test |
|----|--------|---------|-----|-----------------|
| D1 | Disk safety was not revalidated before destroying the target | A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review | `Installer::revalidate_target_disk` re-probes and compares identity in `storage_prepare` | `orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}` |
| D2 | `probe_disks` parsed human-readable `lsblk` sizes as bytes | Every disk displayed `0 B`; the size-identity check was blind | pass `-b` to `lsblk` | `probe::tests::probe_disks_classifies_optical_as_protected` asserts the byte size |
| D3 | The bridge could replay or overlap a destructive install | A second `start_install` (retry, second frontend, reconnect) could re-run storage preparation | approval is single-use, consumed before work; `installing`/`installed` guards | `start_install_cannot_be_replayed_after_success` |
| D4 | The welcome's primary action was not keyboard-focused | No focus ring; Return did not open the installer | request activation + `forceActiveFocus()`; regression test also rejects the invalid `onActiveChanged` signal | `welcome_primary_action_is_keyboard_reachable` |
| D5 | Escape during installation silently closed the frontend | A destructive operation could be abandoned with no explanation | Escape is refused during progress and the UI says so | `progress_cannot_be_abandoned_silently` |
| D6 | Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist | A config-supplied endpoint could target `file:`, inject curl config, or return a huge body | `validate_endpoint`, `proto = "=http,https"`, `max-filesize`, redirects disabled | `endpoint_validation_rejects_unsafe_schemes_and_injection`, `curl_transport_refuses_an_unsafe_endpoint`, `curl_transport_bounds_the_response_and_disables_redirects` |
| D7 | `TargetRoot` followed symlinks out of the target | A planted symlink could redirect an installer write to the host | `reject_symlink_components` in `write`/`ensure_dir` | `target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}` |
| D8 | The VM harness documented disk safety but did not enforce it, and reported observations rather than results | A mistyped `DISK` could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion | `vmsafety.py`; per-run OVMF/QMP; structured results; argument arrays | `test_vmsafety.py` (16 tests), run by `make test` |
## Suspected risks (reproduced or bounded)
- **Prompt injection in device labels / model output.** Typed proposals pass
`deny_unknown_fields` and a closed action set; device labels are display-only.
Bounded by tests (`phoebus-ai` parse/allowlist). No VM canary run.
- **Live sudoers is `NOPASSWD: ALL` (live-only).** It does not reach the
installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
- **OpenCode `/config/providers` returns provider credentials in cleartext.**
Upstream behaviour; the probe keeps only counts and docs mandate loopback +
tunnel. Not a Phoebus defect, but a deployment hazard.
## Missing verification (blocked / not run)
- Full graphical installation past the AI step and two fresh-disk graphical
installs: **BLOCKED** — no provider key and no local model; normal install
requires a connected model.
- `install`/`boot` modes on the final ISO: **NOT RUN** (the earlier appliance
build passed `INSTALL-RC=0` + detached boot; the final ISO's binary changed,
so this is not automatically re-verified).
- Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat,
NVIDIA: **NOT RUN / unverified**.
- Second physical computer OpenCode transport: **BLOCKED**; host-local forwarded
hop and unauthenticated-LAN refusal were tested instead.